mirror of
https://github.com/openglow-org/forgefirm.git
synced 2026-09-27 16:51:12 -07:00
Release acceptance gate: release.sh refuses to sign without a matching artifact
scripts/acceptance-gate.py recomputes every catalog test's domain fingerprint from /etc/forgefirm-manifest.json inside the release rootfs and requires the committed releases/v<version>/acceptance.json to carry a matching PASS (inherited results not core and newer than the invalidate epoch; the artifact self-hashed; the catalog identical to the tree). release.sh runs it after the build and stages the artifact as a release asset; FORGEFIRM_ACCEPTANCE_SKIP=1 bypasses loudly. scripts/manifest-from-tree.py builds the same manifest from the recipe pins with git for CI and the workstation; forgetest-ci.yml runs the unit tests and enforces the coverage lint (every manifest path covered by some test). docs/ACCEPTANCE.md is the contract; the coverage currency rule and the status live in BRINGUP.
This commit is contained in:
ScottW514
@@ -0,0 +1,177 @@
|
||||
# Release acceptance
|
||||
|
||||
A ForgeFIRM release is signed and published only when the **acceptance
|
||||
catalog** passes on the bench machine and the result is committed with
|
||||
the release. This document is the contract: what the gate is, how a
|
||||
campaign runs, how a result stays valid across builds, and what the
|
||||
release pipeline checks.
|
||||
|
||||
## The pieces
|
||||
|
||||
| Piece | Where | What it does |
|
||||
|---|---|---|
|
||||
| **forgetest** | `forgetest/` in this repo; on the **dev image** as a daemon on HTTP **:8090** | Runs the catalog against the machine from a self-contained web page, keeps the append-only result log under `/data/forgetest/`, exports the release artifact, and carries the bench diagnostics page. Never on a release image. |
|
||||
| **Image manifest** | `/etc/forgefirm-manifest.json` in every image (`meta-forgefirm/classes/forgefirm-manifest.bbclass`, `forgefirm-image-manifest.bbclass`) | The build's inputs: for every component the pinned revision and one `[path, blob-id]` pair per source file, plus the platform identity (machine, kernel revision + config hash, device tree hashes, layer content hashes). |
|
||||
| **Artifact** | `releases/v<version>/acceptance.json` (+ `acceptance.md`) committed to this repo, and attached to the GitHub release | What forgetest exported: per catalog test the winning PASS, the fingerprint it ran under, and whether it was inherited. Self-hashed. |
|
||||
| **Gate** | `scripts/acceptance-gate.py`, called by `scripts/release.sh` | Recomputes every test's fingerprint from the manifest inside the release rootfs and requires the recorded PASS to match. |
|
||||
|
||||
## The catalog
|
||||
|
||||
Every test declares, in code (`forgetest/forgetest/suite/*.py`):
|
||||
|
||||
- **kind** - `auto` (no operator), `operator` (prompts, no emission), or
|
||||
`live` (laser emission possible: the page requires the eye-protection /
|
||||
fire-watch / exhaust acknowledgment, and the physical arm press is
|
||||
required through the controller's normal path - forgetest never touches
|
||||
the laser latch);
|
||||
- **hardware** - `api` (forgectrl and the controller stay up) or
|
||||
`takeover` (forgectrl is stopped for the duration; a marker file makes a
|
||||
crash recoverable at the next start);
|
||||
- **covers** - the source paths whose content the test stands for, as
|
||||
`(component, glob)` pairs;
|
||||
- **requires** - tests that must be satisfied first (the emission tests
|
||||
require the motion and readback tests);
|
||||
- **always** - membership in the **always-required core**, which is run
|
||||
in every campaign and is never inherited: image health, the kernel
|
||||
latch/safety readbacks, and one live emission witness with the
|
||||
armed-window disarm.
|
||||
|
||||
`GET /catalog` on the tool lists the definitions; the page shows them under
|
||||
each test's *details*.
|
||||
|
||||
## Domain fingerprints and inheritance
|
||||
|
||||
A test's **domain fingerprint** is the hash of the `(component, path,
|
||||
blob-id)` triples its coverage globs select in the image manifest, plus the
|
||||
platform identity, plus the hash of the test's own implementation. A PASS
|
||||
recorded under fingerprint F applies to any build whose recomputed
|
||||
fingerprint is F - the same code computes it on the board and in the gate.
|
||||
|
||||
Consequences:
|
||||
|
||||
- A change to a covered file invalidates exactly the tests that cover it.
|
||||
A panel-only change reruns the core plus the panel tests, not the
|
||||
cooling drills.
|
||||
- A platform change (kernel, device tree, a layer's content) invalidates
|
||||
everything.
|
||||
- A change to a test's implementation invalidates that test's earlier
|
||||
passes and no other.
|
||||
- "Touched" is computed from content hashes carried in the image, never
|
||||
declared by hand.
|
||||
|
||||
## Campaigns
|
||||
|
||||
A **campaign** is bound to one image (manifest content hash) and one
|
||||
catalog (catalog hash). The first Start on an image opens one. It stays
|
||||
open until a **FAIL** (or an erroring test), an **invalidate-all**, an
|
||||
explicit **reset**, or a different image or catalog. Reboots into the same
|
||||
image continue it.
|
||||
|
||||
For every test, in order:
|
||||
|
||||
1. a PASS in the open campaign with the current fingerprint satisfies it;
|
||||
2. otherwise, if it is not core, the newest PASS anywhere in the history
|
||||
with the current fingerprint and newer than the last invalidate-all is
|
||||
**inherited** (its origin - run time, image, campaign - is kept and
|
||||
exported);
|
||||
3. otherwise it is **required** (reason: `always`, `never-passed`, or
|
||||
`domain-changed`).
|
||||
|
||||
**Release authorized** = a campaign is open and every catalog test is
|
||||
satisfied. There is no SKIP: a test the bench cannot run means the release
|
||||
cannot be authorized (that is a catalog change, not a skip).
|
||||
|
||||
**Invalidate all** (page footer, reason required) records that the bench
|
||||
itself changed - new tube, driver swap, cable work, a judgment call - and
|
||||
forces a full campaign; nothing before it can be inherited.
|
||||
|
||||
## Running a campaign
|
||||
|
||||
1. Boot the dev image on the bench (`forgefirm-image-dev`), open
|
||||
`http://<machine>:8090/`.
|
||||
2. The banner shows the image, the manifest identity, and *Release
|
||||
authorized*. Tests marked **required** need to run; **inherited** ones
|
||||
do not.
|
||||
3. Start the required tests. `operator` tests ask questions in the run
|
||||
pane; `live` tests need the acknowledgment and the physical arm press;
|
||||
`takeover` tests stop forgectrl for the duration.
|
||||
4. When *Release authorized: YES*, **Export release artifact**, download
|
||||
`acceptance.json` and `acceptance.md`, and commit them as
|
||||
`releases/v<version>/acceptance.json` and `.md`.
|
||||
|
||||
The raw log (`/data/forgetest/results.jsonl`, `Raw log` in the footer) is
|
||||
the bench's own record; the artifact is the release's.
|
||||
|
||||
## The gate
|
||||
|
||||
`scripts/release.sh <version>` builds the release image, reads
|
||||
`/etc/forgefirm-manifest.json` out of the release rootfs and runs
|
||||
|
||||
scripts/acceptance-gate.py releases/v<version>/acceptance.json <manifest>
|
||||
|
||||
which requires: the artifact self-hash intact; `authorized: true`; the
|
||||
catalog in the tree identical to the artifact's; for every test a recorded
|
||||
PASS whose fingerprint equals the one recomputed from the release manifest;
|
||||
inherited results not core and newer than the invalidate epoch. Any
|
||||
problem dies before signing. `FORGEFIRM_ACCEPTANCE_SKIP=1` bypasses the gate
|
||||
deliberately and prints a loud warning; it is never the default. The
|
||||
artifact is staged and attached to the GitHub release next to
|
||||
`forgefirm.fw`.
|
||||
|
||||
Because the dev image and the release image are built from the same tree
|
||||
in one `bitbake` invocation, their manifests share the same identity; a pin
|
||||
bumped after the campaign shows up as a fingerprint mismatch on exactly the
|
||||
tests that cover it.
|
||||
|
||||
## Coverage currency rule
|
||||
|
||||
Every change is evaluated against the catalog, in addition to its unit
|
||||
tests:
|
||||
|
||||
1. does an existing test exercise the changed behavior - if not, add or
|
||||
extend one in the same change;
|
||||
2. does that test's `covers` map name the files touched - if not, widen it
|
||||
in the same change.
|
||||
|
||||
A behavior change with no catalog consequence needs a sentence of
|
||||
justification in the commit message. Coverage gaps are defects: under the
|
||||
domain model an uncovered path lets an inherited PASS stay valid across a
|
||||
change that should have invalidated it.
|
||||
|
||||
The mechanical floor is the coverage lint,
|
||||
|
||||
python3 -m forgetest.coverage --manifest <manifest.json> [--enforce]
|
||||
|
||||
which lists every manifest path no test covers, minus the allowlist of
|
||||
non-behavioral paths in `forgetest/forgetest/coverage.py` (docs, CI, tests,
|
||||
licenses). CI (`forgetest-ci.yml`) runs it on a manifest generated from the
|
||||
recipe pins with `scripts/manifest-from-tree.py` (no Yocto build needed)
|
||||
and fails the job on any uncovered path. On the board, run it against
|
||||
`/etc/forgefirm-manifest.json`. The lint proves a
|
||||
file is *fingerprinted*; whether the test *exercises* the change is the
|
||||
change author's judgment (rule 1).
|
||||
|
||||
## Bench diagnostics page
|
||||
|
||||
The same daemon serves `#bench`: the registry of the bench tools
|
||||
(`scripts/bench`, installed under `/usr/share/forgetest/bench/`), each with
|
||||
its safety class (`dry`, `takeover`, `live`, `scope`), argument form, and
|
||||
last run. A ported tool runs as a subprocess with the output on the page;
|
||||
unported tools are listed with Start disabled. Bench runs are recorded in
|
||||
`/data/forgetest/bench.jsonl` and never enter a campaign.
|
||||
|
||||
## Layout
|
||||
|
||||
forgetest/forgetest/ the package (stdlib only)
|
||||
manifest.py manifest, globs, fingerprints, coverage report
|
||||
catalog.py @test registry, catalog hash
|
||||
campaign.py the rules (pure functions)
|
||||
artifact.py export + gate verification
|
||||
runner.py one run at a time, prompts, abort, takeover
|
||||
server.py / page.py HTTP API + the page (forgectrl's access rules)
|
||||
bench.py / coverage.py bench registry + subprocess runner; the lint
|
||||
suite/ the catalog, one module per subsystem
|
||||
forgetest/tests/ host unit tests (python3 -m unittest discover -s tests)
|
||||
scripts/acceptance-gate.py the gate
|
||||
scripts/manifest-from-tree.py manifest from the recipe pins (CI, workstation)
|
||||
releases/v<version>/ the committed artifacts
|
||||
@@ -1379,6 +1379,69 @@ overrides the IE (DE applied while associated to the US AP), and
|
||||
clearing reverts to the 00 hint. The UI labels the default
|
||||
accordingly ("Automatic — AP country, else World").
|
||||
|
||||
## Release acceptance (forgetest, port 8090)
|
||||
|
||||
The release acceptance tool - the catalog, campaigns, domain
|
||||
fingerprints, inheritance, the always-required core, invalidate-all,
|
||||
the release gate, and the coverage currency rule - is specified in
|
||||
`docs/ACCEPTANCE.md`; the tool lives in `forgetest/` and ships only on
|
||||
the dev image (`forgetest` recipe, `/etc/init.d/forgetest`, HTTP :8090).
|
||||
Status: **code landed 2026-08-15, host-verified and build-verified;
|
||||
bench validation pending - ships with the next full image flash** (the
|
||||
image manifest is an image change: `forgefirm-manifest.bbclass` entries
|
||||
from every component recipe, the kernel and the module through
|
||||
`do_deploy`, assembled by `forgefirm-image-manifest.bbclass` into
|
||||
`/etc/forgefirm-manifest.json`, also deployed next to the image as
|
||||
`*.forgefirm-manifest.json`). Build proof (dev image `20260815191634`,
|
||||
built with the classes): the manifest carries all eight components
|
||||
(forgectrl, grblhal-glowforge with the core submodule's files,
|
||||
forgefirm-app merged from its three recipes, python3-gfhardware,
|
||||
python3-gfutilities, kernel-module-glowforge and linux-fslc through the
|
||||
deploy path, forgetest through the file mode), the DTB hashes and the
|
||||
modules directory, and layer content hashes that are **byte-identical to
|
||||
what `scripts/manifest-from-tree.py` computes on the workstation** - the
|
||||
identity is content-defined, independent of the checkout's commit or
|
||||
dirty state; forgetest is installed at S95 with the bench scripts. Host
|
||||
proof: 44 unit tests (campaign
|
||||
rules, fingerprints, artifact build + gate verification incl. the
|
||||
negative fixtures - tampered artifact, covered-file change, platform
|
||||
change, core inherited, stale invalidate, catalog change, implementation
|
||||
change - and the runner + HTTP API end to end with a fake catalog and a
|
||||
fake bench tool), the tree manifest generated from the recipe pins with
|
||||
`scripts/manifest-from-tree.py` (submodule recursion verified on the
|
||||
grblHAL core), the coverage lint reporting on it, and the gate refusing an
|
||||
empty artifact cleanly; `.github/workflows/forgetest-ci.yml` runs the
|
||||
same and **enforces the coverage lint** (every manifest path is covered:
|
||||
0 uncovered on both the built manifest and the tree manifest). **Catalog
|
||||
v1 is complete: 24 tests**, every one a port of a proven bench drill or
|
||||
of a bench-verified check, with the recorded pass criteria: the core
|
||||
`image.health`, `kernel.latch-locked-idle`, `kernel.k1-k2`,
|
||||
`kernel.k3-unlock`, `kernel.fire-abu` (GATE A drills as takeover tests;
|
||||
K3 and fire B/U prompt for the lid when `laser_pgood` reports HV good)
|
||||
and `laser.emission-witness` (S400 square, emission peak -> 0, HV rise,
|
||||
M2 job-based disarm, operator confirms the mark); `forgectrl.auth` /
|
||||
`settings-bounds` / `panel-serves`, `logs.tree-tail-export` (sanitized
|
||||
bundle carries no panel token); `motion.pacing`, `jog-roundtrip`,
|
||||
`liveness-probe`, `cancel-abort`, `deadman` (SIGKILL / SIGSTOP->underrun
|
||||
/ forgectrl restart mid-move, head returned by the kernel counters);
|
||||
`cooling.flow-verify` (through forgectrl's diag runner) and
|
||||
`fans-quiet-after-motion`; `laser.disarm-in-hold`, `expected-stop`
|
||||
(POST /controller/stop mid-burn, then the operator-judged restart),
|
||||
`kill-mid-fire`; `camera.snapshot`; `update.slots-and-signature`;
|
||||
`cloud.mode-switch` (gfcloud comes up and records its service probe) and
|
||||
`cloud.gfhome-homing`. Not in the catalog by design: the stale-origin
|
||||
refusal after an underrun (config-dependent - GRBL mode permits unhomed
|
||||
cutting, see the campaign notes above). The bench tab lists every
|
||||
`scripts/bench` tool; runnable from the page: `check-pwm`,
|
||||
`pacing-test`, `bench-m2`, `bench-phase2`, `cp-watchdog`, `accel-fast`,
|
||||
`bump-seek`, `fire-test`, `gate-a-kernel`, `platform-drills`,
|
||||
`flow-confirm`, `flow-sampler` (takeover tools get forgectrl stopped and
|
||||
started around the run); the scope tools, the host-side flow
|
||||
characterization tools, and the live drills stay ssh/host-run for now.
|
||||
The coverage currency rule is in `CLAUDE.md`
|
||||
"Working rules". Bench validation and the bench-tab ports are Next work
|
||||
item 15.
|
||||
|
||||
## Hardware facts bank (measured)
|
||||
|
||||
- **DRV8825 stepper drivers wedge on 40 V rail glitches** (factory board;
|
||||
@@ -2643,3 +2706,21 @@ accordingly ("Automatic — AP country, else World").
|
||||
goes with the wrapper; a forced daemon crash logs the wrapper's
|
||||
`exited (N) - respawning in 5 s` line under `forgectrl`.
|
||||
|
||||
15. **Release acceptance tool (forgetest) - CODE-COMPLETE 2026-08-15,
|
||||
host- and build-verified; bench validation pending, ships with the
|
||||
next full image flash.** Contract: `docs/ACCEPTANCE.md`; catalog v1
|
||||
complete (24 tests, coverage lint enforced in CI, rule in
|
||||
`CLAUDE.md`). **Images for the flash are archived under
|
||||
`images/20260815193946/`** (release `…193946` + dev `…194415`, one
|
||||
tree; the two manifests share the acceptance identity, the release
|
||||
image carries no forgetest). Remaining, in order: (a) bench: boot
|
||||
that dev image, run the catalog from `:8090` - the takeover, motion,
|
||||
cooling, live
|
||||
and cloud tests are ports of proven scripts and need their first run
|
||||
on the machine (expect pass-criteria tuning: fan tach tolerance,
|
||||
snapshot size floor, timeouts) - export, and drive one UI-only pin
|
||||
bump to prove the inherited/required split; (b) the remaining
|
||||
bench-tab ports (scope tools, host-side flow characterization, the
|
||||
live drills - the catalog carries their acceptance forms); (c) the
|
||||
first release runs the full
|
||||
campaign and commits `releases/v<version>/acceptance.json`.
|
||||
|
||||
@@ -148,8 +148,13 @@ demonstrably untouched.*
|
||||
command (`--publish` runs it where gh is authenticated). Gates:
|
||||
clean tree, version single-source, rootfs-vs-slot size
|
||||
(warn ≥ 170 MiB / fail ≥ 195 MiB, under bitbake's own hard cap),
|
||||
**installer-embedded pubkey must match the signing key**, and
|
||||
factory-era fwup (0.14.2) verification of the packed archive.
|
||||
**installer-embedded pubkey must match the signing key**,
|
||||
factory-era fwup (0.14.2) verification of the packed archive, and the
|
||||
**release acceptance gate**: `releases/v<version>/acceptance.json`
|
||||
(exported by forgetest on the bench) must authorize the built rootfs
|
||||
per `docs/ACCEPTANCE.md` - the gate recomputes every catalog test's
|
||||
domain fingerprint from `/etc/forgefirm-manifest.json` inside the
|
||||
release ext4. The artifact is attached to the GitHub release.
|
||||
- One version source: `FORGEFIRM_RELEASE` = git tag =
|
||||
`/etc/forgefirm-version` = `.fw` meta-version; the script enforces
|
||||
agreement.
|
||||
|
||||
Reference in New Issue
Block a user