gitea: configure registry package cleanup rules to cap per-host CI image churn #8

New Issue

2026-05-23T15:50:41-07:00

vh commented

2026-05-23 15:50:41 -07:00

Motivation

CI/CD pipelines on vh/skaldsong + vh/Worldtree push a new SHA-pinned image per commit. Per-host docker layer caches accumulate indefinitely because nothing GCs them. Snapshot 2026-05-23:

Host	Disk used	Reclaimable	Top churn
ana-docker	205G / 245G (89%)	96.2 GB	47× worldtree, 40× skaldsong
corviduo-dev	50G / 75G (70%)	42.0 GB	27× worldtree

Operator's reasoning during the cleanup: docker-side rollback has never been used in production. Git rollback + re-deploy IS the recovery path. So per-host image retention is dead weight; the durable hedge belongs at the source (gitea registry).

Approach

Gitea v1.26.1 (current on ana-docker) supports native Package Registry Cleanup Rules (per-owner per-type):

Admin UI: Site Administration → Packages → Cleanup Rules, OR per-owner Settings → Packages → Cleanup Rules
API: /owners/{owner}/cleanup_rules (POST/PATCH/DELETE)
GC scheduler: [cron.cleanup_packages] in app.ini — verify enabled + schedule (default daily)

Proposed rule shape:

Owner: vh
Package type: container (Docker registry)
Keep last N versions per repo (suggest N=10) OR delete versions older than X days (suggest 30d)
Verify GC actually drops blob bytes from disk after cleanup

Open questions

Per-repo vs per-owner rule granularity — does v1.26's UI support per-repo overrides, or only per-owner default? (Affects whether vh/Worldtree can have higher N than vh/skaldsong.)
Default branch tags (:latest, :main) — does the rule protect moving-tag images, or do we need explicit exclusions?
Pinned-deployment SHAs — corviduo-dev has THREE active worldtree deployments (demo/personal/pinned) on different SHAs. If the gitea rule deletes a SHA still referenced by a pinned deploy, that host's local cache is fine but re-pull would fail. Need coordination with worldtree-dev on minimum retention floor that covers all live pins.
Cron retention vs ad-hoc cleanup — operator wanted nightly automation as a separate decision. Both can coexist (per-host docker image prune -a --force nightly + gitea-side rule).

Why this is the durable hedge

A per-host backup image (whether "keep last N tags" or --until=Xh) only helps when registry is unreachable. Registry is in-fleet, so that's basically "gitea down" — at which point the whole CD chain is broken anyway. Centralizing retention at gitea means any host can re-pull historical SHAs without each host maintaining its own cache.

Acceptance

A cleanup rule visible in Site Admin or vh owner settings for container packages with a defined retention policy.
Verified cron schedule firing GC (check log lines from cron.cleanup_packages).
After a CI build that creates a new SHA, the oldest SHA(s) past the retention threshold are demonstrably absent from the registry (tea packages list ... or curl /v2/vh/<repo>/tags/list).
A note added to persistent-memory.md capturing the chosen N / age cap so we don't re-debate next quarter.

Related context

2026-05-23 disk snapshot driving this (the unfiltered docker image prune -a that ran on ana-docker + corviduo-dev).
2026-05-19 persistent-memory entry on docker rmi vs prune (the per-host equivalent of this gitea-side fix).

## Motivation CI/CD pipelines on `vh/skaldsong` + `vh/Worldtree` push a new SHA-pinned image per commit. Per-host docker layer caches accumulate indefinitely because nothing GCs them. Snapshot 2026-05-23: | Host | Disk used | Reclaimable | Top churn | |---|---|---|---| | ana-docker | 205G / 245G (89%) | 96.2 GB | 47× worldtree, 40× skaldsong | | corviduo-dev | 50G / 75G (70%) | 42.0 GB | 27× worldtree | Operator's reasoning during the cleanup: docker-side rollback has never been used in production. Git rollback + re-deploy IS the recovery path. So per-host image retention is dead weight; the durable hedge belongs at the **source** (gitea registry). ## Approach Gitea v1.26.1 (current on `ana-docker`) supports native **Package Registry Cleanup Rules** (per-owner per-type): - Admin UI: Site Administration → Packages → Cleanup Rules, OR per-owner Settings → Packages → Cleanup Rules - API: `/owners/{owner}/cleanup_rules` (POST/PATCH/DELETE) - GC scheduler: `[cron.cleanup_packages]` in app.ini — verify enabled + schedule (default daily) Proposed rule shape: - Owner: `vh` - Package type: `container` (Docker registry) - Keep last N versions per repo (suggest N=10) **OR** delete versions older than X days (suggest 30d) - Verify GC actually drops blob bytes from disk after cleanup ## Open questions 1. Per-repo vs per-owner rule granularity — does v1.26's UI support per-repo overrides, or only per-owner default? (Affects whether `vh/Worldtree` can have higher N than `vh/skaldsong`.) 2. Default branch tags (`:latest`, `:main`) — does the rule protect moving-tag images, or do we need explicit exclusions? 3. Pinned-deployment SHAs — corviduo-dev has THREE active worldtree deployments (demo/personal/pinned) on different SHAs. If the gitea rule deletes a SHA still referenced by a pinned deploy, that host's local cache is fine but re-pull would fail. Need coordination with worldtree-dev on minimum retention floor that covers all live pins. 4. Cron retention vs ad-hoc cleanup — operator wanted nightly automation as a separate decision. Both can coexist (per-host `docker image prune -a --force` nightly + gitea-side rule). ## Why this is the durable hedge A per-host backup image (whether "keep last N tags" or `--until=Xh`) only helps when registry is unreachable. Registry is in-fleet, so that's basically "gitea down" — at which point the whole CD chain is broken anyway. Centralizing retention at gitea means any host can re-pull historical SHAs without each host maintaining its own cache. ## Acceptance - A cleanup rule visible in Site Admin or `vh` owner settings for container packages with a defined retention policy. - Verified cron schedule firing GC (check log lines from `cron.cleanup_packages`). - After a CI build that creates a new SHA, the oldest SHA(s) past the retention threshold are demonstrably absent from the registry (`tea packages list ...` or `curl /v2/vh/<repo>/tags/list`). - A note added to `persistent-memory.md` capturing the chosen N / age cap so we don't re-debate next quarter. ## Related context - 2026-05-23 disk snapshot driving this (the unfiltered `docker image prune -a` that ran on ana-docker + corviduo-dev). - 2026-05-19 persistent-memory entry on docker rmi vs prune (the per-host equivalent of this gitea-side fix).

vh added the enhancement label 2026-05-23 15:50:41 -07:00

vh referenced this issue

2026-05-23 18:03:48 -07:00

CD-driven version-tag mirror + per-release cleanup (supersedes #8) #9

vh commented

2026-05-23 18:03:58 -07:00

Superseded by #9 — CD-driven version-tag mirror + per-release cleanup. The keep_pattern-based gitea cleanup rules in this issue assumed versioned tags would exist in the registry; investigation 2026-05-23 found they don't (every push gets only :sha). #9 fixes the upstream tagging problem AND introduces offsite durability for releases, replacing this approach entirely.

vh closed this issue

2026-05-23 18:03:58 -07:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: vh/esh-pfi-infrastructure#8