mirror of
https://github.com/openglow-org/forgefirm.git
synced 2026-09-28 01:01:12 -07:00
ScottW514
0258268aae
A campaign is mostly waiting for the next Start. The page now offers two queues, and each takes every test of its kinds the campaign does not already count as satisfied: Unattended for the auto tests, which need nobody in the room, and Operator and live for the ones that need somebody at the machine, since they prompt and they fire the laser. The buttons say how many they would run and ask before starting; the live queue names the tests that fire and takes the acknowledgment once, for all of them. A queue runs one test at a time through the runner's single slot, in prerequisite order. Registration order otherwise, so a run reads down the page, but a prerequisite inside the queue always goes first. It stops on the first result that is not a PASS: a FAIL closes the campaign, and carrying on would only open a second one behind the operator's back. A test the runner refuses to start is skipped with the reason on the page and the rest carry on, which is what happens to an auto test waiting on an operator one: run the attended queue, then the unattended one again. The queue lives in the runner, not in the tab, so reloading the page or closing it leaves the run alone. While one is up it holds the machine between its tests as well as during them, so a single Start and the bench tools are refused rather than cutting in. Stop the queue cancels what is still waiting and lets the run in progress finish; Abort ends that one too, and lands as the non-PASS that stops the queue. Every run a queue starts records which one put it there. Also moves the /state ETag test to the end of its class. It invalidates, timestamps are whole seconds, and a PASS stamped in the same second as an invalidate is deliberately not inheritable, so on a fast run it decided the inheritance an earlier test was checking. forgetest is a dev-only component and can never appear in a coverage map, so this has no acceptance catalog consequence.
281 lines
16 KiB
Markdown
281 lines
16 KiB
Markdown
# Release acceptance
|
|
|
|
A ForgeFIRM release is signed and published only when the **acceptance
|
|
catalog** passes on the bench machine and the result is committed with
|
|
the release. This document is the contract: what the gate is, how a
|
|
campaign runs, how a result stays valid across builds, and what the
|
|
release pipeline checks.
|
|
|
|
## The pieces
|
|
|
|
| Piece | Where | What it does |
|
|
|---|---|---|
|
|
| **forgetest** | `forgetest/` in this repo; on the **dev image** as a daemon on HTTP **:8090** | Runs the catalog against the machine from a self-contained web page, keeps the append-only result log under `/data/forgetest/`, exports the release artifact, and carries the bench diagnostics page. Never on a release image. |
|
|
| **Image manifest** | `/etc/forgefirm-manifest.json` in every image (`meta-forgefirm/classes/forgefirm-manifest.bbclass`, `forgefirm-image-manifest.bbclass`) | The build's inputs: for every component the pinned revision and one `[path, blob-id]` pair per source file, plus the platform identity (machine, kernel revision + config hash, device tree hashes, layer content hashes). |
|
|
| **Artifact** | `releases/v<version>/acceptance.json` (+ `acceptance.md`) committed to this repo, and attached to the GitHub release | What forgetest exported: per catalog test the winning PASS, the fingerprint it ran under, and whether it was inherited. Self-hashed. |
|
|
| **Gate** | `scripts/acceptance-gate.py`, called by `scripts/release.sh` | Recomputes every test's fingerprint from the manifest inside the release rootfs and requires the recorded PASS to match. |
|
|
|
|
## The catalog
|
|
|
|
Every test declares, in code (`forgetest/forgetest/suite/*.py`):
|
|
|
|
- **kind** - `auto` (no operator), `operator` (prompts, no emission), or
|
|
`live` (laser emission possible: the page requires the eye-protection /
|
|
fire-watch / exhaust acknowledgment, and the physical arm press is
|
|
required through the controller's normal path - forgetest never touches
|
|
the laser latch);
|
|
- **hardware** - `api` (forgectrl and the controller stay up) or
|
|
`takeover` (forgectrl is stopped for the duration; a marker file makes a
|
|
crash recoverable at the next start);
|
|
- **covers** - the source paths whose content the test stands for, as
|
|
`(component, glob)` pairs;
|
|
- **requires** - tests that must be satisfied first (the emission tests
|
|
require the motion and readback tests). This orders the runs; it is not
|
|
a release condition of its own (the release needs every test satisfied
|
|
anyway). The page's **Ignore prerequisites** switch lets any test start
|
|
alone; a run started that way records the unmet prerequisites in its
|
|
`evidence.prerequisites` and its log, and the prerequisites stay
|
|
required;
|
|
- **always** - membership in the **always-required core**, which is run
|
|
in every campaign and is never inherited: image health, the kernel
|
|
latch/safety readbacks, and one live emission witness with the
|
|
armed-window disarm.
|
|
|
|
`GET /catalog` on the tool lists the definitions; the page shows them under
|
|
each test's *details*.
|
|
|
|
## Domain fingerprints and inheritance
|
|
|
|
A test's **domain fingerprint** is the hash of the `(component, path,
|
|
blob-id)` triples its coverage globs select in the image manifest, plus the
|
|
platform identity, plus the hash of the test's own implementation. A PASS
|
|
recorded under fingerprint F applies to any build whose recomputed
|
|
fingerprint is F - the same code computes it on the board and in the gate.
|
|
|
|
Consequences:
|
|
|
|
- A change to a covered file invalidates exactly the tests that cover it.
|
|
A panel-only change reruns the core plus the panel tests, not the
|
|
cooling drills.
|
|
- A platform change (kernel, device tree, a layer's content) invalidates
|
|
everything. Layer content is every file under `meta-forgefirm`,
|
|
`meta-glowforge-bsp` and `meta-openglow-core` except documentation
|
|
(`*.md`) and the component pin files (`<recipe>-pin.inc`, holding only a
|
|
component's `SRCREV` and the `PV` that moves with it). A pin bump is the
|
|
component's change, and the component entry already carries it file by
|
|
file, so it invalidates the tests that cover the component - not the
|
|
bench. A recipe-body change (build flags, patches, config fragments,
|
|
init scripts, a third-party pin with no manifest entry) is layer content
|
|
and invalidates everything; so does a pin written into a recipe body
|
|
instead of its pin file (the safe direction).
|
|
- A change to a test's implementation invalidates that test's earlier
|
|
passes and no other.
|
|
- "Touched" is computed from content hashes carried in the image, never
|
|
declared by hand.
|
|
|
|
## Campaigns
|
|
|
|
A **campaign** is bound to one image (manifest content hash) and one
|
|
catalog (catalog hash). The first Start on an image opens one. It stays
|
|
open until a **FAIL** (or an erroring test), an **invalidate-all**, an
|
|
explicit **reset**, or a different image or catalog. Reboots into the same
|
|
image continue it.
|
|
|
|
For every test, in order:
|
|
|
|
1. a PASS in the open campaign with the current fingerprint satisfies it;
|
|
2. otherwise, if it is not core, the newest PASS anywhere in the history
|
|
with the current fingerprint and newer than the last invalidate-all is
|
|
**inherited** (its origin - run time, image, campaign - is kept and
|
|
exported);
|
|
3. otherwise it is **required** (reason: `always`, `never-passed`, or
|
|
`domain-changed`).
|
|
|
|
**Release authorized** = a campaign is open and every catalog test is
|
|
satisfied. There is no SKIP: a test the bench cannot run means the release
|
|
cannot be authorized (that is a catalog change, not a skip).
|
|
|
|
**Invalidate all** (page footer, reason required) records that the bench
|
|
itself changed - new tube, driver swap, cable work, a judgment call - and
|
|
forces a full campaign; nothing before it can be inherited.
|
|
|
|
**Inheritance is local.** The history a bench inherits from is its own
|
|
results log; there is no import of a published `acceptance.json`. A second
|
|
bench, or one whose `/data` has been wiped, starts from a full campaign.
|
|
|
|
## Running a campaign
|
|
|
|
1. Boot the dev image on the bench (`forgefirm-image-dev`), open
|
|
`http://<machine>:8090/`.
|
|
2. The banner shows the image, the manifest identity, and *Release
|
|
authorized*. Tests marked **required** need to run; **inherited** ones
|
|
do not.
|
|
3. Start the required tests. `operator` tests ask questions in the run
|
|
pane; `live` tests need the acknowledgment and the physical arm press;
|
|
`takeover` tests stop forgectrl for the duration. A test whose
|
|
prerequisites are not satisfied is locked until they are - or until
|
|
the **Ignore prerequisites** switch in the Campaign card is on, which
|
|
unlocks every Start (the switch is remembered by the browser; a run
|
|
started under it says so in its record).
|
|
The `cloud.*` job tests run **in cloud mode and stay there**: the first
|
|
one switches from GRBL mode (once, its connect-time hunt waited out)
|
|
and the following ones reuse the live session; nothing switches back -
|
|
switch on the control panel when done. `cloud.mode-switch` is the one
|
|
round trip and starts in GRBL mode.
|
|
4. Or hand the whole list to a queue. **Run what is left** offers two:
|
|
**Unattended** takes every `auto` test the campaign does not already
|
|
count as satisfied, and needs nobody in the room; **Operator and live**
|
|
takes the `operator` and `live` ones, and needs somebody at the
|
|
machine, since it prompts and it fires the laser. Each button says how
|
|
many it would run, and asks before it starts: the live queue names the
|
|
tests that fire and takes the acknowledgment once, for all of them.
|
|
A queue runs one test at a time in prerequisite order and stops on the
|
|
first result that is not a PASS, because a FAIL closes the campaign. A
|
|
test it cannot start is skipped with the reason on the page and the
|
|
rest carry on, which is what happens to an `auto` test waiting on an
|
|
`operator` one: run the attended queue, then the unattended one again.
|
|
**Stop the queue** cancels what is still waiting and lets the run in
|
|
progress finish; **Abort** ends that one too. The queue lives in the
|
|
runner, so closing the page or reloading it does not disturb the run.
|
|
5. When *Release authorized: YES*, **Export release artifact**, download
|
|
`acceptance.json` and `acceptance.md`, and commit them as
|
|
`releases/v<version>/acceptance.json` and `.md`.
|
|
|
|
The raw log (`/data/forgetest/results.jsonl`, `Raw log` in the footer) is
|
|
the bench's own record; the artifact is the release's.
|
|
|
|
### Every run starts from, and leaves, the fresh-boot idle state
|
|
|
|
The runner brackets every test and bench tool with a **baseline** pass
|
|
(`baseline.py`): before the run it verifies the machine against the
|
|
fresh-boot idle state and restores anything off it; after the run - on
|
|
every exit path, pass, fail, or abort - it restores again. Two kinds of
|
|
items: **fixed** resting values the boot establishes (the kernel module
|
|
defaults, forgectrl's start-up writes, the GRBL controller's init writes:
|
|
`motor_lock=8`, `x/y_mode=8`, `x/y_decay=1`, `step_freq=28160`,
|
|
`ramp_rate=125000`, `streaming=0`, `state=idle`, latch locked, hold
|
|
currents, head lamp and button LEDs off, heater and TEC off, the lid lamp
|
|
at forgectrl's `lid_lamp_idle` setting; forgectrl: the controller running
|
|
with motion verified, no diagnostic, the camera engine and cooling engine
|
|
idle), and **preserved** state with no resting policy that a run must
|
|
hand back as it found it (the position counters, the settings map, the
|
|
controller mode). The mode in force decides what the baseline owns: in
|
|
cloud mode the cloud client's own configuration (the GRBL controller's
|
|
init values, which it rewrites from every pulse header; the lid lamp,
|
|
its lid-image level; the position counters, re-zeroed at every service
|
|
action) is left to it, and the safety readbacks, latch, ring, module
|
|
defaults, and forgectrl's engines are checked as always. The mode itself
|
|
is preserved unless the run declared the change (`ctx.mode_changed()`,
|
|
the cloud tests entering cloud mode); the persisted `controller_mode`
|
|
setting is never written back as a bare setting - only the switch keeps
|
|
it in step with the live mode. Deviations are
|
|
**leftovers**: logged in the run pane, kept in the result's `evidence`
|
|
(`baseline.pre` / `baseline.post`), and surfaced in the page's message
|
|
line - a leftover found before a run is attributed to the previous run; one
|
|
found after is the run's own defect. Takeover runs additionally capture
|
|
the controller-owned kernel attributes on entry and write them back before
|
|
forgectrl restarts, so the supervisor's liveness probe runs on the machine
|
|
it expects. The runner waits for forgectrl's supervisor to settle (motion
|
|
verified, or the ladder's verdict) before and after every takeover.
|
|
|
|
**Power-cycle before a campaign.** forgetest takes a **fresh-boot
|
|
reference** once per boot (`/data/forgetest/boot-<boot_id>.json`, taken
|
|
only within the first ten minutes after boot, after the supervisor
|
|
settles): the whole idle picture of this machine as the image boots it,
|
|
the check on the fixed values, and the record a leftover is judged
|
|
against. Take it after a **power cycle**, not a warm `reboot` - the
|
|
machine's true fresh state is the powered-on one (the PIC's own lamp and
|
|
sensor defaults, then forgectrl's start-up writes on top).
|
|
A displaced head is jogged back along its own path by the kernel-measured
|
|
X/Y delta (bounded to 100 mm; Z is never touched); beyond that the
|
|
counters are reported and the run must be fixed. A run that legitimately
|
|
re-zeroes the counters (cloud mode's connect) tells the runner so
|
|
(`ctx.counters_rezeroed()`) and hands the head back itself.
|
|
|
|
## The gate
|
|
|
|
`scripts/release.sh <version>` builds the release image, reads
|
|
`/etc/forgefirm-manifest.json` out of the release rootfs and runs
|
|
|
|
scripts/acceptance-gate.py releases/v<version>/acceptance.json <manifest>
|
|
|
|
which requires: the artifact self-hash intact; `authorized: true`; the
|
|
catalog in the tree identical to the artifact's; for every test a recorded
|
|
PASS whose fingerprint equals the one recomputed from the release manifest;
|
|
inherited results not core and newer than the invalidate epoch. Any
|
|
problem dies before signing. `FORGEFIRM_ACCEPTANCE_SKIP=1` bypasses the gate
|
|
deliberately and prints a loud warning; it is never the default. The
|
|
artifact is staged and attached to the GitHub release next to
|
|
`forgefirm.fw`.
|
|
|
|
Because the dev image and the release image are built from the same tree
|
|
in one `bitbake` invocation, their manifests share the same identity; a pin
|
|
bumped after the campaign shows up as a fingerprint mismatch on exactly the
|
|
tests that cover it.
|
|
|
|
## Coverage currency rule
|
|
|
|
Every change is evaluated against the catalog, in addition to its unit
|
|
tests:
|
|
|
|
1. does an existing test exercise the changed behavior - if not, add or
|
|
extend one in the same change;
|
|
2. does that test's `covers` map name the files touched - if not, widen it
|
|
in the same change.
|
|
|
|
A behavior change with no catalog consequence needs a sentence of
|
|
justification in the commit message. Coverage gaps are defects: under the
|
|
domain model an uncovered path lets an inherited PASS stay valid across a
|
|
change that should have invalidated it.
|
|
|
|
The mechanical floor is the coverage lint,
|
|
|
|
python3 -m forgetest.coverage --manifest <manifest.json> [--enforce]
|
|
|
|
which lists every manifest path no test covers, minus the allowlist of
|
|
non-behavioral paths in `forgetest/forgetest/coverage.py` (docs, CI, tests,
|
|
licenses). CI (`forgetest-ci.yml`) runs it on a manifest generated from the
|
|
recipe pins with `scripts/manifest-from-tree.py` (no Yocto build needed)
|
|
and fails the job on any uncovered path. On the board, run it against
|
|
`/etc/forgefirm-manifest.json`. The lint proves a
|
|
file is *fingerprinted*; whether the test *exercises* the change is the
|
|
change author's judgment (rule 1).
|
|
|
|
## Bench diagnostics page
|
|
|
|
The same daemon serves `#bench`: the registry of the bench tools
|
|
(`scripts/bench`, installed under `/usr/share/forgetest/bench/`), each with
|
|
its safety class, argument form, and last run. The classes: `dry` (reads
|
|
or dry motion, forgectrl stays up), `takeover` (forgectrl and the
|
|
controller stopped for the run, the pulse device free, the same wrapper
|
|
the takeover tests use), `scope` (a takeover whose result only means
|
|
something with the named instrument on the bench), `live` (emission
|
|
possible; the operator acknowledgment and the physical arm press). A tool
|
|
runs as a subprocess with the output on the page and, on the board, the
|
|
machine as `GF_HOST=127.0.0.1`, the panel token in `GF_TOKEN`, and its data
|
|
files under `/data/forgetest/bench/` (`FORGETEST_BENCH_DATA`); the same
|
|
scripts run from a LAN host with `GF_HOST` set (`scripts/bench/gfbench.py`,
|
|
`scripts/bench/README.md`). Every board-runnable tool is ported; the entries
|
|
that stay unported are the CI harnesses of the null-sink controller and the
|
|
factory `.puls` decoder, which do not run against the machine at all - they
|
|
are listed so the catalog of what exists is complete. Bench runs are
|
|
recorded in `/data/forgetest/bench.jsonl` and never enter a campaign.
|
|
|
|
## Layout
|
|
|
|
forgetest/forgetest/ the package (stdlib only)
|
|
manifest.py manifest, globs, fingerprints, coverage report
|
|
catalog.py @test registry, catalog hash
|
|
campaign.py the rules (pure functions)
|
|
artifact.py export + gate verification
|
|
runner.py one run at a time, prompts, abort, takeover, queues
|
|
baseline.py the fresh-boot idle state around every run
|
|
server.py / page.py HTTP API + the page (forgectrl's access rules)
|
|
bench.py / coverage.py bench registry + subprocess runner; the lint
|
|
suite/ the catalog, one module per subsystem
|
|
forgetest/tests/ host unit tests (python3 -m unittest discover -s tests)
|
|
scripts/bench/ the bench tools (+ gfbench.py, the board/host helper)
|
|
scripts/acceptance-gate.py the gate
|
|
scripts/manifest-from-tree.py manifest from the recipe pins (CI, workstation)
|
|
releases/v<version>/ the committed artifacts
|