mirror of
https://github.com/openglow-org/forgefirm.git
synced 2026-09-28 01:01:12 -07:00
ScottW514
05d68ba0a8
BRINGUP had grown to 3,122 lines in which the same subject was answered differently depending on where the reader stopped: GATE A "stays open, no live-fire" in the phase text and closed in the campaign record, the catalog at 24 tests in one section and 35 in another, several "bench validation pending" headings over bodies that recorded the pass. Split by kind rather than by age. BRINGUP (907 lines) is the present state only - status, bench runbook, laser, lid/interlock/button policy, homing, forgectrl, diagnostics, logging, release acceptance, the measured facts bank, and a Next work list of the 15 items that are actually open. CAMPAIGN-LOG (2,657 lines) takes the dated blocks verbatim, in chronological order, and is append-only: a correction is a later entry, not an edit. Its two reading rules are stated up front, since moved text keeps its original "above"/"below" and its pre-split item numbers. Facts corrected against the tree while rewriting: the liveness probe gates at p2p 800, not 500, with the real wedge, noise and jolt figures; the panel has seven tabs including Logs and is built from src/ui/, not ui.c; the devserver replaced tools/mock.py; /status reports real head presence; the estop_halts_motion opt-in is gone; core PR #999 is merged, leaving only the step_us_min commit fork-only; the update system stands at Phase 5, the uSDHC pads and the 2026-08-08 kernel batch have shipped, and 8 MP camera capture is tracked here for the first time. SAFETY and UPDATE-SYSTEM point their drill records at the log. ACCEPTANCE records that inheritance is local: there is no import of a published artifact, so a second bench starts from a full campaign - one of the four open items the retired tool plan held, the rest of which are now in BRINGUP's acceptance item. Documentation only, no behavior change and no catalog consequence: docs/ is outside every layer and .md is excluded from the layer content hash.
266 lines
15 KiB
Markdown
266 lines
15 KiB
Markdown
# Release acceptance
|
|
|
|
A ForgeFIRM release is signed and published only when the **acceptance
|
|
catalog** passes on the bench machine and the result is committed with
|
|
the release. This document is the contract: what the gate is, how a
|
|
campaign runs, how a result stays valid across builds, and what the
|
|
release pipeline checks.
|
|
|
|
## The pieces
|
|
|
|
| Piece | Where | What it does |
|
|
|---|---|---|
|
|
| **forgetest** | `forgetest/` in this repo; on the **dev image** as a daemon on HTTP **:8090** | Runs the catalog against the machine from a self-contained web page, keeps the append-only result log under `/data/forgetest/`, exports the release artifact, and carries the bench diagnostics page. Never on a release image. |
|
|
| **Image manifest** | `/etc/forgefirm-manifest.json` in every image (`meta-forgefirm/classes/forgefirm-manifest.bbclass`, `forgefirm-image-manifest.bbclass`) | The build's inputs: for every component the pinned revision and one `[path, blob-id]` pair per source file, plus the platform identity (machine, kernel revision + config hash, device tree hashes, layer content hashes). |
|
|
| **Artifact** | `releases/v<version>/acceptance.json` (+ `acceptance.md`) committed to this repo, and attached to the GitHub release | What forgetest exported: per catalog test the winning PASS, the fingerprint it ran under, and whether it was inherited. Self-hashed. |
|
|
| **Gate** | `scripts/acceptance-gate.py`, called by `scripts/release.sh` | Recomputes every test's fingerprint from the manifest inside the release rootfs and requires the recorded PASS to match. |
|
|
|
|
## The catalog
|
|
|
|
Every test declares, in code (`forgetest/forgetest/suite/*.py`):
|
|
|
|
- **kind** - `auto` (no operator), `operator` (prompts, no emission), or
|
|
`live` (laser emission possible: the page requires the eye-protection /
|
|
fire-watch / exhaust acknowledgment, and the physical arm press is
|
|
required through the controller's normal path - forgetest never touches
|
|
the laser latch);
|
|
- **hardware** - `api` (forgectrl and the controller stay up) or
|
|
`takeover` (forgectrl is stopped for the duration; a marker file makes a
|
|
crash recoverable at the next start);
|
|
- **covers** - the source paths whose content the test stands for, as
|
|
`(component, glob)` pairs;
|
|
- **requires** - tests that must be satisfied first (the emission tests
|
|
require the motion and readback tests). This orders the runs; it is not
|
|
a release condition of its own (the release needs every test satisfied
|
|
anyway). The page's **Ignore prerequisites** switch lets any test start
|
|
alone; a run started that way records the unmet prerequisites in its
|
|
`evidence.prerequisites` and its log, and the prerequisites stay
|
|
required;
|
|
- **always** - membership in the **always-required core**, which is run
|
|
in every campaign and is never inherited: image health, the kernel
|
|
latch/safety readbacks, and one live emission witness with the
|
|
armed-window disarm.
|
|
|
|
`GET /catalog` on the tool lists the definitions; the page shows them under
|
|
each test's *details*.
|
|
|
|
## Domain fingerprints and inheritance
|
|
|
|
A test's **domain fingerprint** is the hash of the `(component, path,
|
|
blob-id)` triples its coverage globs select in the image manifest, plus the
|
|
platform identity, plus the hash of the test's own implementation. A PASS
|
|
recorded under fingerprint F applies to any build whose recomputed
|
|
fingerprint is F - the same code computes it on the board and in the gate.
|
|
|
|
Consequences:
|
|
|
|
- A change to a covered file invalidates exactly the tests that cover it.
|
|
A panel-only change reruns the core plus the panel tests, not the
|
|
cooling drills.
|
|
- A platform change (kernel, device tree, a layer's content) invalidates
|
|
everything. Layer content is every file under `meta-forgefirm`,
|
|
`meta-glowforge-bsp` and `meta-openglow-core` except documentation
|
|
(`*.md`) and the component pin files (`<recipe>-pin.inc`, holding only a
|
|
component's `SRCREV` and the `PV` that moves with it). A pin bump is the
|
|
component's change, and the component entry already carries it file by
|
|
file, so it invalidates the tests that cover the component - not the
|
|
bench. A recipe-body change (build flags, patches, config fragments,
|
|
init scripts, a third-party pin with no manifest entry) is layer content
|
|
and invalidates everything; so does a pin written into a recipe body
|
|
instead of its pin file (the safe direction).
|
|
- A change to a test's implementation invalidates that test's earlier
|
|
passes and no other.
|
|
- "Touched" is computed from content hashes carried in the image, never
|
|
declared by hand.
|
|
|
|
## Campaigns
|
|
|
|
A **campaign** is bound to one image (manifest content hash) and one
|
|
catalog (catalog hash). The first Start on an image opens one. It stays
|
|
open until a **FAIL** (or an erroring test), an **invalidate-all**, an
|
|
explicit **reset**, or a different image or catalog. Reboots into the same
|
|
image continue it.
|
|
|
|
For every test, in order:
|
|
|
|
1. a PASS in the open campaign with the current fingerprint satisfies it;
|
|
2. otherwise, if it is not core, the newest PASS anywhere in the history
|
|
with the current fingerprint and newer than the last invalidate-all is
|
|
**inherited** (its origin - run time, image, campaign - is kept and
|
|
exported);
|
|
3. otherwise it is **required** (reason: `always`, `never-passed`, or
|
|
`domain-changed`).
|
|
|
|
**Release authorized** = a campaign is open and every catalog test is
|
|
satisfied. There is no SKIP: a test the bench cannot run means the release
|
|
cannot be authorized (that is a catalog change, not a skip).
|
|
|
|
**Invalidate all** (page footer, reason required) records that the bench
|
|
itself changed - new tube, driver swap, cable work, a judgment call - and
|
|
forces a full campaign; nothing before it can be inherited.
|
|
|
|
**Inheritance is local.** The history a bench inherits from is its own
|
|
results log; there is no import of a published `acceptance.json`. A second
|
|
bench, or one whose `/data` has been wiped, starts from a full campaign.
|
|
|
|
## Running a campaign
|
|
|
|
1. Boot the dev image on the bench (`forgefirm-image-dev`), open
|
|
`http://<machine>:8090/`.
|
|
2. The banner shows the image, the manifest identity, and *Release
|
|
authorized*. Tests marked **required** need to run; **inherited** ones
|
|
do not.
|
|
3. Start the required tests. `operator` tests ask questions in the run
|
|
pane; `live` tests need the acknowledgment and the physical arm press;
|
|
`takeover` tests stop forgectrl for the duration. A test whose
|
|
prerequisites are not satisfied is locked until they are - or until
|
|
the **Ignore prerequisites** switch in the Campaign card is on, which
|
|
unlocks every Start (the switch is remembered by the browser; a run
|
|
started under it says so in its record).
|
|
The `cloud.*` job tests run **in cloud mode and stay there**: the first
|
|
one switches from GRBL mode (once, its connect-time hunt waited out)
|
|
and the following ones reuse the live session; nothing switches back -
|
|
switch on the control panel when done. `cloud.mode-switch` is the one
|
|
round trip and starts in GRBL mode.
|
|
4. When *Release authorized: YES*, **Export release artifact**, download
|
|
`acceptance.json` and `acceptance.md`, and commit them as
|
|
`releases/v<version>/acceptance.json` and `.md`.
|
|
|
|
The raw log (`/data/forgetest/results.jsonl`, `Raw log` in the footer) is
|
|
the bench's own record; the artifact is the release's.
|
|
|
|
### Every run starts from, and leaves, the fresh-boot idle state
|
|
|
|
The runner brackets every test and bench tool with a **baseline** pass
|
|
(`baseline.py`): before the run it verifies the machine against the
|
|
fresh-boot idle state and restores anything off it; after the run - on
|
|
every exit path, pass, fail, or abort - it restores again. Two kinds of
|
|
items: **fixed** resting values the boot establishes (the kernel module
|
|
defaults, forgectrl's start-up writes, the GRBL controller's init writes:
|
|
`motor_lock=8`, `x/y_mode=8`, `x/y_decay=1`, `step_freq=28160`,
|
|
`ramp_rate=125000`, `streaming=0`, `state=idle`, latch locked, hold
|
|
currents, head lamp and button LEDs off, heater and TEC off, the lid lamp
|
|
at forgectrl's `lid_lamp_idle` setting; forgectrl: the controller running
|
|
with motion verified, no diagnostic, the camera engine and cooling engine
|
|
idle), and **preserved** state with no resting policy that a run must
|
|
hand back as it found it (the position counters, the settings map, the
|
|
controller mode). The mode in force decides what the baseline owns: in
|
|
cloud mode the cloud client's own configuration (the GRBL controller's
|
|
init values, which it rewrites from every pulse header; the lid lamp,
|
|
its lid-image level; the position counters, re-zeroed at every service
|
|
action) is left to it, and the safety readbacks, latch, ring, module
|
|
defaults, and forgectrl's engines are checked as always. The mode itself
|
|
is preserved unless the run declared the change (`ctx.mode_changed()`,
|
|
the cloud tests entering cloud mode); the persisted `controller_mode`
|
|
setting is never written back as a bare setting - only the switch keeps
|
|
it in step with the live mode. Deviations are
|
|
**leftovers**: logged in the run pane, kept in the result's `evidence`
|
|
(`baseline.pre` / `baseline.post`), and surfaced in the page's message
|
|
line - a leftover found before a run is attributed to the previous run; one
|
|
found after is the run's own defect. Takeover runs additionally capture
|
|
the controller-owned kernel attributes on entry and write them back before
|
|
forgectrl restarts, so the supervisor's liveness probe runs on the machine
|
|
it expects. The runner waits for forgectrl's supervisor to settle (motion
|
|
verified, or the ladder's verdict) before and after every takeover.
|
|
|
|
**Power-cycle before a campaign.** forgetest takes a **fresh-boot
|
|
reference** once per boot (`/data/forgetest/boot-<boot_id>.json`, taken
|
|
only within the first ten minutes after boot, after the supervisor
|
|
settles): the whole idle picture of this machine as the image boots it,
|
|
the check on the fixed values, and the record a leftover is judged
|
|
against. Take it after a **power cycle**, not a warm `reboot` - the
|
|
machine's true fresh state is the powered-on one (the PIC's own lamp and
|
|
sensor defaults, then forgectrl's start-up writes on top).
|
|
A displaced head is jogged back along its own path by the kernel-measured
|
|
X/Y delta (bounded to 100 mm; Z is never touched); beyond that the
|
|
counters are reported and the run must be fixed. A run that legitimately
|
|
re-zeroes the counters (cloud mode's connect) tells the runner so
|
|
(`ctx.counters_rezeroed()`) and hands the head back itself.
|
|
|
|
## The gate
|
|
|
|
`scripts/release.sh <version>` builds the release image, reads
|
|
`/etc/forgefirm-manifest.json` out of the release rootfs and runs
|
|
|
|
scripts/acceptance-gate.py releases/v<version>/acceptance.json <manifest>
|
|
|
|
which requires: the artifact self-hash intact; `authorized: true`; the
|
|
catalog in the tree identical to the artifact's; for every test a recorded
|
|
PASS whose fingerprint equals the one recomputed from the release manifest;
|
|
inherited results not core and newer than the invalidate epoch. Any
|
|
problem dies before signing. `FORGEFIRM_ACCEPTANCE_SKIP=1` bypasses the gate
|
|
deliberately and prints a loud warning; it is never the default. The
|
|
artifact is staged and attached to the GitHub release next to
|
|
`forgefirm.fw`.
|
|
|
|
Because the dev image and the release image are built from the same tree
|
|
in one `bitbake` invocation, their manifests share the same identity; a pin
|
|
bumped after the campaign shows up as a fingerprint mismatch on exactly the
|
|
tests that cover it.
|
|
|
|
## Coverage currency rule
|
|
|
|
Every change is evaluated against the catalog, in addition to its unit
|
|
tests:
|
|
|
|
1. does an existing test exercise the changed behavior - if not, add or
|
|
extend one in the same change;
|
|
2. does that test's `covers` map name the files touched - if not, widen it
|
|
in the same change.
|
|
|
|
A behavior change with no catalog consequence needs a sentence of
|
|
justification in the commit message. Coverage gaps are defects: under the
|
|
domain model an uncovered path lets an inherited PASS stay valid across a
|
|
change that should have invalidated it.
|
|
|
|
The mechanical floor is the coverage lint,
|
|
|
|
python3 -m forgetest.coverage --manifest <manifest.json> [--enforce]
|
|
|
|
which lists every manifest path no test covers, minus the allowlist of
|
|
non-behavioral paths in `forgetest/forgetest/coverage.py` (docs, CI, tests,
|
|
licenses). CI (`forgetest-ci.yml`) runs it on a manifest generated from the
|
|
recipe pins with `scripts/manifest-from-tree.py` (no Yocto build needed)
|
|
and fails the job on any uncovered path. On the board, run it against
|
|
`/etc/forgefirm-manifest.json`. The lint proves a
|
|
file is *fingerprinted*; whether the test *exercises* the change is the
|
|
change author's judgment (rule 1).
|
|
|
|
## Bench diagnostics page
|
|
|
|
The same daemon serves `#bench`: the registry of the bench tools
|
|
(`scripts/bench`, installed under `/usr/share/forgetest/bench/`), each with
|
|
its safety class, argument form, and last run. The classes: `dry` (reads
|
|
or dry motion, forgectrl stays up), `takeover` (forgectrl and the
|
|
controller stopped for the run, the pulse device free, the same wrapper
|
|
the takeover tests use), `scope` (a takeover whose result only means
|
|
something with the named instrument on the bench), `live` (emission
|
|
possible; the operator acknowledgment and the physical arm press). A tool
|
|
runs as a subprocess with the output on the page and, on the board, the
|
|
machine as `GF_HOST=127.0.0.1`, the panel token in `GF_TOKEN`, and its data
|
|
files under `/data/forgetest/bench/` (`FORGETEST_BENCH_DATA`); the same
|
|
scripts run from a LAN host with `GF_HOST` set (`scripts/bench/gfbench.py`,
|
|
`scripts/bench/README.md`). Every board-runnable tool is ported; the entries
|
|
that stay unported are the CI harnesses of the null-sink controller and the
|
|
factory `.puls` decoder, which do not run against the machine at all - they
|
|
are listed so the catalog of what exists is complete. Bench runs are
|
|
recorded in `/data/forgetest/bench.jsonl` and never enter a campaign.
|
|
|
|
## Layout
|
|
|
|
forgetest/forgetest/ the package (stdlib only)
|
|
manifest.py manifest, globs, fingerprints, coverage report
|
|
catalog.py @test registry, catalog hash
|
|
campaign.py the rules (pure functions)
|
|
artifact.py export + gate verification
|
|
runner.py one run at a time, prompts, abort, takeover
|
|
baseline.py the fresh-boot idle state around every run
|
|
server.py / page.py HTTP API + the page (forgectrl's access rules)
|
|
bench.py / coverage.py bench registry + subprocess runner; the lint
|
|
suite/ the catalog, one module per subsystem
|
|
forgetest/tests/ host unit tests (python3 -m unittest discover -s tests)
|
|
scripts/bench/ the bench tools (+ gfbench.py, the board/host helper)
|
|
scripts/acceptance-gate.py the gate
|
|
scripts/manifest-from-tree.py manifest from the recipe pins (CI, workstation)
|
|
releases/v<version>/ the committed artifacts
|