Disk pressure from CI image churn: each commit to vh/skaldsong, vh/Worldtree, etc. pushes a new SHA-pinned image; they accumulate indefinitely. Snapshot 2026-05-23: ana-docker at 89% disk, 96 GB of reclaimable gitea registry blobs. Manually pruned (~113 GB recovered) but the pattern recurs.
Loss of release-grade durability: operator wants every versioned commit (semver-tagged) preserved indefinitely, but today every push gets ONLY a :<40-char-sha> tag — there are zero :v0.27.0-style tags in the registry. The registry cannot distinguish "release worth preserving" from "ephemeral SHA churn." Issue #8 proposed gitea-side cleanup rules with a keep-pattern for ^v[0-9]+\.[0-9]+\.[0-9]+$, but the inputs that pattern matches don't exist yet.
Issue #8 attacked symptom 1 (disk pressure) with per-owner cleanup rules and assumed version tags would be available as the protected set. With versioned tags absent from the registry, those rules can't do their intended job — they'd protect latest and nothing else, deleting everything between releases. We need to fix the upstream tagging problem AND introduce offsite durability for releases. The CD-driven model below does both atomically, replacing the gitea-side rules entirely.
Proposed shape: CD does the work on every release
When CI builds a commit that has a semver git tag, the same pipeline:
Multi-tag the build: push :<sha> + :v<version> + :latest in one buildx invocation.
Mirror the versioned tag offsite: crane copy gitea.phasefinal.com/vh/<repo>:v<x> $MIRROR/<repo>:v<x>.
Prune the gitea registry of non-protected tags: keep :v*, :latest, and the current :<sha>; delete the rest via gitea's DELETE /api/v1/packages/{owner}/container/{name}/{version} API.
Non-versioned pushes (regular main commits) still publish :<sha> for CD's pull-and-recreate, but skip steps 2-3. Accumulation between releases is bounded by release cadence.
Sketch — real version needs pagination, retries, mirror-down handling.
DRY: reusable workflow
To avoid copying the 30-line block into N consumer repos, factor as a reusable workflow at vh/ci-shared/.gitea/workflows/release-mirror.yaml that each consumer repo uses: from its own CI. One source of truth; one place to fix bugs / add features.
For the operator's fleet (4-6 Bifrost-consumer repos, all theirs, all on Gitea Actions), CD-driven is the cleaner shape.
Open questions
Offsite mirror target. Lean: Distribution registry (the OCI reference) on nh3-docker, NFS-backed to nh3-nas. Lightweight, OCI-compliant, crane/docker push works natively. Alternatives: second Gitea instance (heavyweight), ghcr.io (free for public, $ for private), Cloudflare R2, OR crane pull to OCI tarballs on nh3-nas (no registry — restorability needs crane push from tarball later).
Mirror-side cleanup: do versioned tags on the mirror live forever or also age out? "Preserve every versioned commit" implies forever. ~200 MB/image × ~weekly releases × N repos = ~10 GB/year/repo. Trivial for SAN-class storage.
Mirror auth: per-repo tokens (better blast radius) vs shared MIRROR_TOKEN secret (single rotation point). Likely shared for v1, per-repo if a compromise scare materializes.
Mirror push failures and CD coupling: should mirror+prune block the deploy step, or run in a needs: deploy post-deploy job so a flaky deploy doesn't leave a half-state? Lean: post-deploy job (atomicity per release-as-a-whole, not per-step).
Per-repo gitea token scope for the prune step: needs write:package at minimum (the existing tea token in this repo only has write:misc,write:notification,write:organization,write:issue,write:repository,write:user — no package scope). Mint a CI-only token with just read:package,write:package scope.
Sequencing
Pick the offsite target (operator decision; lean: Distribution registry on nh3-docker → nh3-nas backing).
Stand up the mirror registry as a new stack in stacks/registry-mirror/. Infra-ops work.
Author the reusable workflow at vh/ci-shared/.gitea/workflows/release-mirror.yaml. Could be infra-ops or skaldsong-dev — neutral.
Per-repo CI updates: each consumer repo (skaldsong, Worldtree, mead-hall, althing-chamber, ratatoskr) adds the if: startsWith(github.ref, 'refs/tags/v') block invoking the shared workflow. One small PR per repo, owned by that repo's dev.
Mint a CI-scoped gitea token with package read+write, distribute as GITEA_TOKEN secret to each consumer repo's Gitea Actions config.
A versioned commit on any consumer repo results in: :v<x> + :latest + :<sha> in gitea; :v<x> in the mirror; all prior non-protected tags for that repo removed from gitea.
Mirror survives the gitea host being completely rebuilt — versioned releases are pull-able from the mirror.
Disk usage on ana-docker for gitea registry blobs trends flat over time rather than growing.
Driven by 2026-05-23 disk snapshot (89% on ana-docker) and the subsequent discovery that no versioned tags exist in the registry today.
## Motivation
Two coupled problems this addresses:
1. **Disk pressure from CI image churn**: each commit to `vh/skaldsong`, `vh/Worldtree`, etc. pushes a new SHA-pinned image; they accumulate indefinitely. Snapshot 2026-05-23: ana-docker at 89% disk, 96 GB of reclaimable gitea registry blobs. Manually pruned (~113 GB recovered) but the pattern recurs.
2. **Loss of release-grade durability**: operator wants every versioned commit (semver-tagged) preserved indefinitely, but today every push gets ONLY a `:<40-char-sha>` tag — there are zero `:v0.27.0`-style tags in the registry. The registry cannot distinguish "release worth preserving" from "ephemeral SHA churn." Issue #8 proposed gitea-side cleanup rules with a keep-pattern for `^v[0-9]+\.[0-9]+\.[0-9]+$`, but the inputs that pattern matches don't exist yet.
## Why supersede #8
Issue #8 attacked symptom 1 (disk pressure) with per-owner cleanup rules and assumed version tags would be available as the protected set. With versioned tags absent from the registry, those rules can't do their intended job — they'd protect `latest` and nothing else, deleting everything between releases. We need to fix the upstream tagging problem AND introduce offsite durability for releases. The CD-driven model below does both atomically, replacing the gitea-side rules entirely.
## Proposed shape: CD does the work on every release
When CI builds a commit that has a semver git tag, the same pipeline:
1. **Multi-tag the build**: push `:<sha>` + `:v<version>` + `:latest` in one buildx invocation.
2. **Mirror the versioned tag offsite**: `crane copy gitea.phasefinal.com/vh/<repo>:v<x> $MIRROR/<repo>:v<x>`.
3. **Prune the gitea registry of non-protected tags**: keep `:v*`, `:latest`, and the current `:<sha>`; delete the rest via gitea's `DELETE /api/v1/packages/{owner}/container/{name}/{version}` API.
Non-versioned pushes (regular `main` commits) still publish `:<sha>` for CD's pull-and-recreate, but skip steps 2-3. Accumulation between releases is bounded by release cadence.
Workflow sketch (each consumer repo's `.gitea/workflows/deploy.yaml`):
```yaml
- name: Push image with version + latest + sha tags
if: startsWith(github.ref, 'refs/tags/v')
run: |
VER=${GITHUB_REF#refs/tags/}
docker buildx build \
--tag $REG/$REPO:${{ github.sha }} \
--tag $REG/$REPO:$VER \
--tag $REG/$REPO:latest \
--push .
- name: Mirror version tag to offsite registry
if: startsWith(github.ref, 'refs/tags/v')
run: |
VER=${GITHUB_REF#refs/tags/}
crane copy $REG/$REPO:$VER $MIRROR_REG/$REPO:$VER
- name: Prune old SHA-only tags from gitea registry
if: startsWith(github.ref, 'refs/tags/v')
run: |
VER=${GITHUB_REF#refs/tags/}
KEEP="${VER}|latest|${{ github.sha }}|^v[0-9]+\\.[0-9]+\\.[0-9]+$"
curl -sf -H "Authorization: token ${{ secrets.GITEA_TOKEN }}" \
"$REG_API/packages/$OWNER?type=container&q=$REPO" \
| jq -r '.[] | .version' \
| grep -vE "$KEEP" \
| xargs -I{} curl -X DELETE -H "Authorization: token ${{ secrets.GITEA_TOKEN }}" \
"$REG_API/packages/$OWNER/container/$REPO/{}"
```
Sketch — real version needs pagination, retries, mirror-down handling.
## DRY: reusable workflow
To avoid copying the 30-line block into N consumer repos, factor as a reusable workflow at `vh/ci-shared/.gitea/workflows/release-mirror.yaml` that each consumer repo `uses:` from its own CI. One source of truth; one place to fix bugs / add features.
## Trade-offs vs scheduled cron approach
| | CD-driven (this) | Scheduled mirror cron |
|---|---|---|
| Atomicity | Per-release atomic | Eventual consistency |
| Where logic lives | Per consumer repo (factored via shared workflow) | infra-ops repo, one place |
| Failure mode if mirror down | CD step soft-fails (configurable) | Next cron tick retries |
| Tracks "what is a release" | Definitionally — the tag IS the trigger | Has to discover via tag-name regex |
| Replaces #8's gitea cleanup rules | Yes — the prune step IS the cleanup | No, both needed |
For the operator's fleet (4-6 Bifrost-consumer repos, all theirs, all on Gitea Actions), CD-driven is the cleaner shape.
## Open questions
1. **Offsite mirror target.** Lean: Distribution registry (the OCI reference) on nh3-docker, NFS-backed to nh3-nas. Lightweight, OCI-compliant, `crane`/`docker` push works natively. Alternatives: second Gitea instance (heavyweight), `ghcr.io` (free for public, $ for private), Cloudflare R2, OR `crane pull` to OCI tarballs on nh3-nas (no registry — restorability needs `crane push` from tarball later).
2. **Mirror-side cleanup**: do versioned tags on the mirror live forever or also age out? "Preserve every versioned commit" implies forever. ~200 MB/image × ~weekly releases × N repos = ~10 GB/year/repo. Trivial for SAN-class storage.
3. **Mirror auth**: per-repo tokens (better blast radius) vs shared `MIRROR_TOKEN` secret (single rotation point). Likely shared for v1, per-repo if a compromise scare materializes.
4. **Mirror push failures and CD coupling**: should mirror+prune block the deploy step, or run in a `needs: deploy` post-deploy job so a flaky deploy doesn't leave a half-state? Lean: post-deploy job (atomicity per release-as-a-whole, not per-step).
5. **Per-repo gitea token scope** for the prune step: needs `write:package` at minimum (the existing tea token in this repo only has `write:misc,write:notification,write:organization,write:issue,write:repository,write:user` — no package scope). Mint a CI-only token with just `read:package,write:package` scope.
## Sequencing
1. **Pick the offsite target** (operator decision; lean: Distribution registry on nh3-docker → nh3-nas backing).
2. **Stand up the mirror registry** as a new stack in `stacks/registry-mirror/`. Infra-ops work.
3. **Author the reusable workflow** at `vh/ci-shared/.gitea/workflows/release-mirror.yaml`. Could be infra-ops or skaldsong-dev — neutral.
4. **Per-repo CI updates**: each consumer repo (skaldsong, Worldtree, mead-hall, althing-chamber, ratatoskr) adds the `if: startsWith(github.ref, 'refs/tags/v')` block invoking the shared workflow. One small PR per repo, owned by that repo's dev.
5. **Mint a CI-scoped gitea token** with package read+write, distribute as `GITEA_TOKEN` secret to each consumer repo's Gitea Actions config.
6. **Close #8** as superseded once this lands.
## Acceptance
- A versioned commit on any consumer repo results in: `:v<x>` + `:latest` + `:<sha>` in gitea; `:v<x>` in the mirror; all prior non-protected tags for that repo removed from gitea.
- Mirror survives the gitea host being completely rebuilt — versioned releases are pull-able from the mirror.
- Disk usage on ana-docker for gitea registry blobs trends flat over time rather than growing.
## Related
- Supersedes #8.
- Driven by 2026-05-23 disk snapshot (89% on ana-docker) and the subsequent discovery that no versioned tags exist in the registry today.
vh
added the enhancement label 2026-05-23 18:03:48 -07:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Motivation
Two coupled problems this addresses:
vh/skaldsong,vh/Worldtree, etc. pushes a new SHA-pinned image; they accumulate indefinitely. Snapshot 2026-05-23: ana-docker at 89% disk, 96 GB of reclaimable gitea registry blobs. Manually pruned (~113 GB recovered) but the pattern recurs.:<40-char-sha>tag — there are zero:v0.27.0-style tags in the registry. The registry cannot distinguish "release worth preserving" from "ephemeral SHA churn." Issue #8 proposed gitea-side cleanup rules with a keep-pattern for^v[0-9]+\.[0-9]+\.[0-9]+$, but the inputs that pattern matches don't exist yet.Why supersede #8
Issue #8 attacked symptom 1 (disk pressure) with per-owner cleanup rules and assumed version tags would be available as the protected set. With versioned tags absent from the registry, those rules can't do their intended job — they'd protect
latestand nothing else, deleting everything between releases. We need to fix the upstream tagging problem AND introduce offsite durability for releases. The CD-driven model below does both atomically, replacing the gitea-side rules entirely.Proposed shape: CD does the work on every release
When CI builds a commit that has a semver git tag, the same pipeline:
:<sha>+:v<version>+:latestin one buildx invocation.crane copy gitea.phasefinal.com/vh/<repo>:v<x> $MIRROR/<repo>:v<x>.:v*,:latest, and the current:<sha>; delete the rest via gitea'sDELETE /api/v1/packages/{owner}/container/{name}/{version}API.Non-versioned pushes (regular
maincommits) still publish:<sha>for CD's pull-and-recreate, but skip steps 2-3. Accumulation between releases is bounded by release cadence.Workflow sketch (each consumer repo's
.gitea/workflows/deploy.yaml):Sketch — real version needs pagination, retries, mirror-down handling.
DRY: reusable workflow
To avoid copying the 30-line block into N consumer repos, factor as a reusable workflow at
vh/ci-shared/.gitea/workflows/release-mirror.yamlthat each consumer repouses:from its own CI. One source of truth; one place to fix bugs / add features.Trade-offs vs scheduled cron approach
For the operator's fleet (4-6 Bifrost-consumer repos, all theirs, all on Gitea Actions), CD-driven is the cleaner shape.
Open questions
Offsite mirror target. Lean: Distribution registry (the OCI reference) on nh3-docker, NFS-backed to nh3-nas. Lightweight, OCI-compliant,
crane/dockerpush works natively. Alternatives: second Gitea instance (heavyweight),ghcr.io(free for public, $ for private), Cloudflare R2, ORcrane pullto OCI tarballs on nh3-nas (no registry — restorability needscrane pushfrom tarball later).Mirror-side cleanup: do versioned tags on the mirror live forever or also age out? "Preserve every versioned commit" implies forever. ~200 MB/image × ~weekly releases × N repos = ~10 GB/year/repo. Trivial for SAN-class storage.
Mirror auth: per-repo tokens (better blast radius) vs shared
MIRROR_TOKENsecret (single rotation point). Likely shared for v1, per-repo if a compromise scare materializes.Mirror push failures and CD coupling: should mirror+prune block the deploy step, or run in a
needs: deploypost-deploy job so a flaky deploy doesn't leave a half-state? Lean: post-deploy job (atomicity per release-as-a-whole, not per-step).Per-repo gitea token scope for the prune step: needs
write:packageat minimum (the existing tea token in this repo only haswrite:misc,write:notification,write:organization,write:issue,write:repository,write:user— no package scope). Mint a CI-only token with justread:package,write:packagescope.Sequencing
stacks/registry-mirror/. Infra-ops work.vh/ci-shared/.gitea/workflows/release-mirror.yaml. Could be infra-ops or skaldsong-dev — neutral.if: startsWith(github.ref, 'refs/tags/v')block invoking the shared workflow. One small PR per repo, owned by that repo's dev.GITEA_TOKENsecret to each consumer repo's Gitea Actions config.Acceptance
:v<x>+:latest+:<sha>in gitea;:v<x>in the mirror; all prior non-protected tags for that repo removed from gitea.Related