mirror of
https://github.com/openglow-org/forgefirm.git
synced 2026-09-27 16:51:12 -07:00
ScottW514
86ce0419e5
The cloud job tests (lid-abort, lid-during-button-wait, hunt-lid-open,
pause-resume) run in cloud mode and leave the machine there: enter_cloud
reuses a live session (pid-scoped from the client's own websocket state
lines) and switches once from GRBL mode, declaring the change to the
baseline; nothing switches back. Each test judges the log from its own
window, the print by its own "print [id]: finished" line, and waits the
service's follow-up moves out (wait_quiet) before it ends. hunt-lid-open
restarts the cloud client through the supervisor's stop/start lever for
a fresh connect. The former switch-back is what failed the last bench run
of hunt-lid-open (409 machine is not idle: the service was still
re-finding the head after the lid closed).
The baseline is mode-aware: in cloud mode the client owns the GRBL
controller's init values, the lid lamp, and the position counters; the
mode itself is preserved unless the run declared the change
(Context.mode_changed); controller_mode is never handed back as a bare
setting (a bare write left the persisted mode out of step with the live
one). The undeclared-change restore now uses the captured state, which
the post pass never saw before.
The acceptance page gets an "Ignore prerequisites" switch (remembered by
the browser): POST /start {ignore_requires} starts a test whose requires
are unmet, and the run records the unmet prerequisites in its evidence
and log; they stay required for the release. The cloud tests' requires
no longer chain through cloud.mode-switch.
Proof: tests/test_cloud_suite.py replays the four tests on the bench's
own gfcloud excerpts (fixtures/) and the run loop's emitted pause/resume
lines under the real runner Context against a fake forgectrl; baseline
mode tests and the server override test; the whole suite (98) and the
coverage lint pass. Catalog consequence: cloud.* fingerprints move with
the module; the catalog hash moves with the requires.
262 lines
14 KiB
Markdown
262 lines
14 KiB
Markdown
# Release acceptance
|
|
|
|
A ForgeFIRM release is signed and published only when the **acceptance
|
|
catalog** passes on the bench machine and the result is committed with
|
|
the release. This document is the contract: what the gate is, how a
|
|
campaign runs, how a result stays valid across builds, and what the
|
|
release pipeline checks.
|
|
|
|
## The pieces
|
|
|
|
| Piece | Where | What it does |
|
|
|---|---|---|
|
|
| **forgetest** | `forgetest/` in this repo; on the **dev image** as a daemon on HTTP **:8090** | Runs the catalog against the machine from a self-contained web page, keeps the append-only result log under `/data/forgetest/`, exports the release artifact, and carries the bench diagnostics page. Never on a release image. |
|
|
| **Image manifest** | `/etc/forgefirm-manifest.json` in every image (`meta-forgefirm/classes/forgefirm-manifest.bbclass`, `forgefirm-image-manifest.bbclass`) | The build's inputs: for every component the pinned revision and one `[path, blob-id]` pair per source file, plus the platform identity (machine, kernel revision + config hash, device tree hashes, layer content hashes). |
|
|
| **Artifact** | `releases/v<version>/acceptance.json` (+ `acceptance.md`) committed to this repo, and attached to the GitHub release | What forgetest exported: per catalog test the winning PASS, the fingerprint it ran under, and whether it was inherited. Self-hashed. |
|
|
| **Gate** | `scripts/acceptance-gate.py`, called by `scripts/release.sh` | Recomputes every test's fingerprint from the manifest inside the release rootfs and requires the recorded PASS to match. |
|
|
|
|
## The catalog
|
|
|
|
Every test declares, in code (`forgetest/forgetest/suite/*.py`):
|
|
|
|
- **kind** - `auto` (no operator), `operator` (prompts, no emission), or
|
|
`live` (laser emission possible: the page requires the eye-protection /
|
|
fire-watch / exhaust acknowledgment, and the physical arm press is
|
|
required through the controller's normal path - forgetest never touches
|
|
the laser latch);
|
|
- **hardware** - `api` (forgectrl and the controller stay up) or
|
|
`takeover` (forgectrl is stopped for the duration; a marker file makes a
|
|
crash recoverable at the next start);
|
|
- **covers** - the source paths whose content the test stands for, as
|
|
`(component, glob)` pairs;
|
|
- **requires** - tests that must be satisfied first (the emission tests
|
|
require the motion and readback tests). This orders the runs; it is not
|
|
a release condition of its own (the release needs every test satisfied
|
|
anyway). The page's **Ignore prerequisites** switch lets any test start
|
|
alone; a run started that way records the unmet prerequisites in its
|
|
`evidence.prerequisites` and its log, and the prerequisites stay
|
|
required;
|
|
- **always** - membership in the **always-required core**, which is run
|
|
in every campaign and is never inherited: image health, the kernel
|
|
latch/safety readbacks, and one live emission witness with the
|
|
armed-window disarm.
|
|
|
|
`GET /catalog` on the tool lists the definitions; the page shows them under
|
|
each test's *details*.
|
|
|
|
## Domain fingerprints and inheritance
|
|
|
|
A test's **domain fingerprint** is the hash of the `(component, path,
|
|
blob-id)` triples its coverage globs select in the image manifest, plus the
|
|
platform identity, plus the hash of the test's own implementation. A PASS
|
|
recorded under fingerprint F applies to any build whose recomputed
|
|
fingerprint is F - the same code computes it on the board and in the gate.
|
|
|
|
Consequences:
|
|
|
|
- A change to a covered file invalidates exactly the tests that cover it.
|
|
A panel-only change reruns the core plus the panel tests, not the
|
|
cooling drills.
|
|
- A platform change (kernel, device tree, a layer's content) invalidates
|
|
everything. Layer content is every file under `meta-forgefirm`,
|
|
`meta-glowforge-bsp` and `meta-openglow-core` except documentation
|
|
(`*.md`) and the component pin files (`<recipe>-pin.inc`, holding only a
|
|
component's `SRCREV` and the `PV` that moves with it). A pin bump is the
|
|
component's change, and the component entry already carries it file by
|
|
file, so it invalidates the tests that cover the component - not the
|
|
bench. A recipe-body change (build flags, patches, config fragments,
|
|
init scripts, a third-party pin with no manifest entry) is layer content
|
|
and invalidates everything; so does a pin written into a recipe body
|
|
instead of its pin file (the safe direction).
|
|
- A change to a test's implementation invalidates that test's earlier
|
|
passes and no other.
|
|
- "Touched" is computed from content hashes carried in the image, never
|
|
declared by hand.
|
|
|
|
## Campaigns
|
|
|
|
A **campaign** is bound to one image (manifest content hash) and one
|
|
catalog (catalog hash). The first Start on an image opens one. It stays
|
|
open until a **FAIL** (or an erroring test), an **invalidate-all**, an
|
|
explicit **reset**, or a different image or catalog. Reboots into the same
|
|
image continue it.
|
|
|
|
For every test, in order:
|
|
|
|
1. a PASS in the open campaign with the current fingerprint satisfies it;
|
|
2. otherwise, if it is not core, the newest PASS anywhere in the history
|
|
with the current fingerprint and newer than the last invalidate-all is
|
|
**inherited** (its origin - run time, image, campaign - is kept and
|
|
exported);
|
|
3. otherwise it is **required** (reason: `always`, `never-passed`, or
|
|
`domain-changed`).
|
|
|
|
**Release authorized** = a campaign is open and every catalog test is
|
|
satisfied. There is no SKIP: a test the bench cannot run means the release
|
|
cannot be authorized (that is a catalog change, not a skip).
|
|
|
|
**Invalidate all** (page footer, reason required) records that the bench
|
|
itself changed - new tube, driver swap, cable work, a judgment call - and
|
|
forces a full campaign; nothing before it can be inherited.
|
|
|
|
## Running a campaign
|
|
|
|
1. Boot the dev image on the bench (`forgefirm-image-dev`), open
|
|
`http://<machine>:8090/`.
|
|
2. The banner shows the image, the manifest identity, and *Release
|
|
authorized*. Tests marked **required** need to run; **inherited** ones
|
|
do not.
|
|
3. Start the required tests. `operator` tests ask questions in the run
|
|
pane; `live` tests need the acknowledgment and the physical arm press;
|
|
`takeover` tests stop forgectrl for the duration. A test whose
|
|
prerequisites are not satisfied is locked until they are - or until
|
|
the **Ignore prerequisites** switch in the Campaign card is on, which
|
|
unlocks every Start (the switch is remembered by the browser; a run
|
|
started under it says so in its record).
|
|
The `cloud.*` job tests run **in cloud mode and stay there**: the first
|
|
one switches from GRBL mode (once, its connect-time hunt waited out)
|
|
and the following ones reuse the live session; nothing switches back -
|
|
switch on the control panel when done. `cloud.mode-switch` is the one
|
|
round trip and starts in GRBL mode.
|
|
4. When *Release authorized: YES*, **Export release artifact**, download
|
|
`acceptance.json` and `acceptance.md`, and commit them as
|
|
`releases/v<version>/acceptance.json` and `.md`.
|
|
|
|
The raw log (`/data/forgetest/results.jsonl`, `Raw log` in the footer) is
|
|
the bench's own record; the artifact is the release's.
|
|
|
|
### Every run starts from, and leaves, the fresh-boot idle state
|
|
|
|
The runner brackets every test and bench tool with a **baseline** pass
|
|
(`baseline.py`): before the run it verifies the machine against the
|
|
fresh-boot idle state and restores anything off it; after the run - on
|
|
every exit path, pass, fail, or abort - it restores again. Two kinds of
|
|
items: **fixed** resting values the boot establishes (the kernel module
|
|
defaults, forgectrl's start-up writes, the GRBL controller's init writes:
|
|
`motor_lock=8`, `x/y_mode=8`, `x/y_decay=1`, `step_freq=28160`,
|
|
`ramp_rate=125000`, `streaming=0`, `state=idle`, latch locked, hold
|
|
currents, head lamp and button LEDs off, heater and TEC off, the lid lamp
|
|
at forgectrl's `lid_lamp_idle` setting; forgectrl: the controller running
|
|
with motion verified, no diagnostic, the camera engine and cooling engine
|
|
idle), and **preserved** state with no resting policy that a run must
|
|
hand back as it found it (the position counters, the settings map, the
|
|
controller mode). The mode in force decides what the baseline owns: in
|
|
cloud mode the cloud client's own configuration (the GRBL controller's
|
|
init values, which it rewrites from every pulse header; the lid lamp,
|
|
its lid-image level; the position counters, re-zeroed at every service
|
|
action) is left to it, and the safety readbacks, latch, ring, module
|
|
defaults, and forgectrl's engines are checked as always. The mode itself
|
|
is preserved unless the run declared the change (`ctx.mode_changed()`,
|
|
the cloud tests entering cloud mode); the persisted `controller_mode`
|
|
setting is never written back as a bare setting - only the switch keeps
|
|
it in step with the live mode. Deviations are
|
|
**leftovers**: logged in the run pane, kept in the result's `evidence`
|
|
(`baseline.pre` / `baseline.post`), and surfaced in the page's message
|
|
line - a leftover found before a run is attributed to the previous run; one
|
|
found after is the run's own defect. Takeover runs additionally capture
|
|
the controller-owned kernel attributes on entry and write them back before
|
|
forgectrl restarts, so the supervisor's liveness probe runs on the machine
|
|
it expects. The runner waits for forgectrl's supervisor to settle (motion
|
|
verified, or the ladder's verdict) before and after every takeover.
|
|
|
|
**Power-cycle before a campaign.** forgetest takes a **fresh-boot
|
|
reference** once per boot (`/data/forgetest/boot-<boot_id>.json`, taken
|
|
only within the first ten minutes after boot, after the supervisor
|
|
settles): the whole idle picture of this machine as the image boots it,
|
|
the check on the fixed values, and the record a leftover is judged
|
|
against. Take it after a **power cycle**, not a warm `reboot` - the
|
|
machine's true fresh state is the powered-on one (the PIC's own lamp and
|
|
sensor defaults, then forgectrl's start-up writes on top).
|
|
A displaced head is jogged back along its own path by the kernel-measured
|
|
X/Y delta (bounded to 100 mm; Z is never touched); beyond that the
|
|
counters are reported and the run must be fixed. A run that legitimately
|
|
re-zeroes the counters (cloud mode's connect) tells the runner so
|
|
(`ctx.counters_rezeroed()`) and hands the head back itself.
|
|
|
|
## The gate
|
|
|
|
`scripts/release.sh <version>` builds the release image, reads
|
|
`/etc/forgefirm-manifest.json` out of the release rootfs and runs
|
|
|
|
scripts/acceptance-gate.py releases/v<version>/acceptance.json <manifest>
|
|
|
|
which requires: the artifact self-hash intact; `authorized: true`; the
|
|
catalog in the tree identical to the artifact's; for every test a recorded
|
|
PASS whose fingerprint equals the one recomputed from the release manifest;
|
|
inherited results not core and newer than the invalidate epoch. Any
|
|
problem dies before signing. `FORGEFIRM_ACCEPTANCE_SKIP=1` bypasses the gate
|
|
deliberately and prints a loud warning; it is never the default. The
|
|
artifact is staged and attached to the GitHub release next to
|
|
`forgefirm.fw`.
|
|
|
|
Because the dev image and the release image are built from the same tree
|
|
in one `bitbake` invocation, their manifests share the same identity; a pin
|
|
bumped after the campaign shows up as a fingerprint mismatch on exactly the
|
|
tests that cover it.
|
|
|
|
## Coverage currency rule
|
|
|
|
Every change is evaluated against the catalog, in addition to its unit
|
|
tests:
|
|
|
|
1. does an existing test exercise the changed behavior - if not, add or
|
|
extend one in the same change;
|
|
2. does that test's `covers` map name the files touched - if not, widen it
|
|
in the same change.
|
|
|
|
A behavior change with no catalog consequence needs a sentence of
|
|
justification in the commit message. Coverage gaps are defects: under the
|
|
domain model an uncovered path lets an inherited PASS stay valid across a
|
|
change that should have invalidated it.
|
|
|
|
The mechanical floor is the coverage lint,
|
|
|
|
python3 -m forgetest.coverage --manifest <manifest.json> [--enforce]
|
|
|
|
which lists every manifest path no test covers, minus the allowlist of
|
|
non-behavioral paths in `forgetest/forgetest/coverage.py` (docs, CI, tests,
|
|
licenses). CI (`forgetest-ci.yml`) runs it on a manifest generated from the
|
|
recipe pins with `scripts/manifest-from-tree.py` (no Yocto build needed)
|
|
and fails the job on any uncovered path. On the board, run it against
|
|
`/etc/forgefirm-manifest.json`. The lint proves a
|
|
file is *fingerprinted*; whether the test *exercises* the change is the
|
|
change author's judgment (rule 1).
|
|
|
|
## Bench diagnostics page
|
|
|
|
The same daemon serves `#bench`: the registry of the bench tools
|
|
(`scripts/bench`, installed under `/usr/share/forgetest/bench/`), each with
|
|
its safety class, argument form, and last run. The classes: `dry` (reads
|
|
or dry motion, forgectrl stays up), `takeover` (forgectrl and the
|
|
controller stopped for the run, the pulse device free, the same wrapper
|
|
the takeover tests use), `scope` (a takeover whose result only means
|
|
something with the named instrument on the bench), `live` (emission
|
|
possible; the operator acknowledgment and the physical arm press). A tool
|
|
runs as a subprocess with the output on the page and, on the board, the
|
|
machine as `GF_HOST=127.0.0.1`, the panel token in `GF_TOKEN`, and its data
|
|
files under `/data/forgetest/bench/` (`FORGETEST_BENCH_DATA`); the same
|
|
scripts run from a LAN host with `GF_HOST` set (`scripts/bench/gfbench.py`,
|
|
`scripts/bench/README.md`). Every board-runnable tool is ported; the entries
|
|
that stay unported are the CI harnesses of the null-sink controller and the
|
|
factory `.puls` decoder, which do not run against the machine at all - they
|
|
are listed so the catalog of what exists is complete. Bench runs are
|
|
recorded in `/data/forgetest/bench.jsonl` and never enter a campaign.
|
|
|
|
## Layout
|
|
|
|
forgetest/forgetest/ the package (stdlib only)
|
|
manifest.py manifest, globs, fingerprints, coverage report
|
|
catalog.py @test registry, catalog hash
|
|
campaign.py the rules (pure functions)
|
|
artifact.py export + gate verification
|
|
runner.py one run at a time, prompts, abort, takeover
|
|
baseline.py the fresh-boot idle state around every run
|
|
server.py / page.py HTTP API + the page (forgectrl's access rules)
|
|
bench.py / coverage.py bench registry + subprocess runner; the lint
|
|
suite/ the catalog, one module per subsystem
|
|
forgetest/tests/ host unit tests (python3 -m unittest discover -s tests)
|
|
scripts/bench/ the bench tools (+ gfbench.py, the board/host helper)
|
|
scripts/acceptance-gate.py the gate
|
|
scripts/manifest-from-tree.py manifest from the recipe pins (CI, workstation)
|
|
releases/v<version>/ the committed artifacts
|