forgefirm/docs/ACCEPTANCE.md

Release acceptance

A ForgeFIRM release is signed and published only when the acceptance catalog passes on the bench machine and the result is committed with the release. This document is the contract: what the gate is, how a campaign runs, how a result stays valid across builds, and what the release pipeline checks.

The pieces

Piece	Where	What it does
forgetest	forgetest/ in this repo; on the dev image as a daemon on HTTP :8090	Runs the catalog against the machine from a self-contained web page, keeps the append-only result log under /data/forgetest/, exports the release artifact, and carries the bench diagnostics page. Never on a release image.
Image manifest	/etc/forgefirm-manifest.json in every image (meta-forgefirm/classes/forgefirm-manifest.bbclass, forgefirm-image-manifest.bbclass)	The build's inputs: for every component the pinned revision and one [path, blob-id] pair per source file, plus the platform identity (machine, kernel revision + config hash, device tree hashes, layer content hashes).
Artifact	releases/v<version>/acceptance.json (+ acceptance.md) committed to this repo, and attached to the GitHub release	What forgetest exported: per catalog test the winning PASS, the fingerprint it ran under, and whether it was inherited. Self-hashed.
Gate	scripts/acceptance-gate.py, called by scripts/release.sh	Recomputes every test's fingerprint from the manifest inside the release rootfs and requires the recorded PASS to match.

The catalog

Every test declares, in code (forgetest/forgetest/suite/*.py):

• kind - auto (no operator), operator (prompts, no emission), or live (laser emission possible: the page requires the eye-protection / fire-watch / exhaust acknowledgment, and the physical arm press is required through the controller's normal path - forgetest never touches the laser latch);
• hardware - api (forgectrl and the controller stay up) or takeover (forgectrl is stopped for the duration; a marker file makes a crash recoverable at the next start);
• mode - the controller mode the test needs live when it starts (grbl or cloud), or none. The runner switches the machine there before the test (through POST /mode, settled and with the Grbl port answering) and leaves it there; a test with no mode runs in whatever mode it finds, or manages the mode itself (the cloud.* job tests, through enter_cloud, which also waits for the service session);
• covers - the source paths whose content the test stands for, as (component, glob) pairs;
• requires - tests that must be satisfied first (the emission tests require the motion and readback tests). This orders the runs; it is not a release condition of its own (the release needs every test satisfied anyway). The page's Ignore prerequisites switch lets any test start alone; a run started that way records the unmet prerequisites in its evidence.prerequisites and its log, and the prerequisites stay required;
• always - membership in the always-required core, which is run in every campaign and is never inherited: image health, the kernel latch/safety readbacks, and one live emission witness with the armed-window disarm.

GET /catalog on the tool lists the definitions; the page shows them under each test's details.

Domain fingerprints and inheritance

A test's domain fingerprint is the hash of the (component, path, blob-id) triples its coverage globs select in the image manifest, plus the platform identity, plus the hash of the test's own implementation. A PASS recorded under fingerprint F applies to any build whose recomputed fingerprint is F - the same code computes it on the board and in the gate.

Consequences:

• A change to a covered file invalidates exactly the tests that cover it. A panel-only change reruns the core plus the panel tests, not the cooling drills.
• A platform change (kernel, device tree, a layer's content) invalidates everything. Layer content is every file under meta-forgefirm, meta-glowforge-bsp and meta-openglow-core except documentation (*.md) and the component pin files (<recipe>-pin.inc, holding only a component's SRCREV and the PV that moves with it). A pin bump is the component's change, and the component entry already carries it file by file, so it invalidates the tests that cover the component - not the bench. A recipe-body change (build flags, patches, config fragments, init scripts, a third-party pin with no manifest entry) is layer content and invalidates everything; so does a pin written into a recipe body instead of its pin file (the safe direction).
• A change to a test's implementation invalidates that test's earlier passes and no other.
• "Touched" is computed from content hashes carried in the image, never declared by hand.

Campaigns

A campaign is bound to one image (manifest content hash) and one catalog (catalog hash). The first Start on an image opens one. It stays open until a FAIL (or an erroring test), an invalidate-all, an explicit reset, or a different image or catalog. Reboots into the same image continue it.

For every test, in order:

1. a PASS in the open campaign with the current fingerprint satisfies it;
2. otherwise, if it is not core, the newest PASS anywhere in the history with the current fingerprint and newer than the last invalidate-all is inherited (its origin - run time, image, campaign - is kept and exported);
3. otherwise it is required (reason: always, never-passed, or domain-changed).

Release authorized = a campaign is open and every catalog test is satisfied. There is no SKIP: a test the bench cannot run means the release cannot be authorized (that is a catalog change, not a skip).

Invalidate all (page footer, reason required) records that the bench itself changed - new tube, driver swap, cable work, a judgment call - and forces a full campaign; nothing before it can be inherited.

Inheritance is local. The history a bench inherits from is its own results log; there is no import of a published acceptance.json. A second bench, or one whose /data has been wiped, starts from a full campaign.

Running a campaign

1. Boot the dev image on the bench (forgefirm-image-dev), open http://<machine>:8090/.
2. The banner shows the image, the manifest identity, and Release authorized. Tests marked required need to run; inherited ones do not.
3. Start the required tests. operator tests ask questions in the run pane; live tests need the acknowledgment and the physical arm press; takeover tests stop forgectrl for the duration. A test whose prerequisites are not satisfied is locked until they are - or until the Ignore prerequisites switch in the Campaign card is on, which unlocks every Start (the switch is remembered by the browser; a run started under it says so in its record). The cloud.* job tests run in cloud mode and stay there: the first one switches from GRBL mode (once, its connect-time hunt waited out) and the following ones reuse the live session; nothing switches back after them. The tests that need GRBL mode (motion.*, laser.*, cooling.fans-quiet-after-motion, cloud.mode-switch, cloud.gfhome-homing) declare it, and the runner switches back the moment one of them starts - so the mode changes only where the next test asks for it, never between tests of the same mode. cloud.mode-switch is the one round trip.
4. Or hand the whole list to a queue. Run what is left offers two: Unattended takes every auto test the campaign does not already count as satisfied, and needs nobody in the room; Operator and live takes the operator and live ones, and needs somebody at the machine, since it prompts and it fires the laser. Each button says how many it would run, and asks before it starts: the live queue names the tests that fire and takes the acknowledgment once, for all of them. A queue runs one test at a time in prerequisite order and stops on the first result that is not a PASS, because a FAIL closes the campaign. A test it cannot start is skipped with the reason on the page and the rest carry on, which is what happens to an auto test waiting on an operator one: run the attended queue, then the unattended one again. Stop the queue cancels what is still waiting and lets the run in progress finish; Abort ends that one too. The queue lives in the runner, so closing the page or reloading it does not disturb the run.
5. When Release authorized: YES, Export release artifact, download acceptance.json and acceptance.md, and commit them as releases/v<version>/acceptance.json and .md.

The raw log (/data/forgetest/results.jsonl, Raw log in the footer) is the bench's own record; the artifact is the release's.

Every run starts from, and leaves, the fresh-boot idle state

The runner brackets every test and bench tool with a baseline pass (baseline.py): before the run it verifies the machine against the fresh-boot idle state and restores anything off it; after the run - on every exit path, pass, fail, or abort - it restores again. Two kinds of items: fixed resting values the boot establishes (the kernel module defaults, forgectrl's start-up writes, the GRBL controller's init writes: motor_lock=8, x/y_mode=8, x/y_decay=1, step_freq=28160, ramp_rate=125000, streaming=0, state=idle, latch locked, hold currents, head lamp and button LEDs off, heater and TEC off, the lid lamp at forgectrl's lid_lamp_idle setting; forgectrl: the controller running with motion verified, no diagnostic, the camera engine and cooling engine idle), and preserved state with no resting policy that a run must hand back as it found it (the position counters, the settings map, the controller mode). The mode in force decides what the baseline owns: in cloud mode the cloud client's own configuration (the GRBL controller's init values, which it rewrites from every pulse header; the lid lamp, its lid-image level; the position counters, re-zeroed at every service action) is left to it, and the safety readbacks, latch, ring, module defaults, and forgectrl's engines are checked as always. The mode itself is preserved unless the run declared the change (ctx.mode_changed(), the cloud tests entering cloud mode) or the test declared a mode, in which case the runner makes the switch in the pre pass, before the preserved state is captured, and the post pass keeps the mode the test asked for; the persisted controller_mode setting is never written back as a bare setting - only the switch keeps it in step with the live mode. Deviations are leftovers: logged in the run pane, kept in the result's evidence (baseline.pre / baseline.post), and surfaced in the page's message line - a leftover found before a run is attributed to the previous run; one found after is the run's own defect. Takeover runs additionally capture the controller-owned kernel attributes on entry and write them back before forgectrl restarts, so the supervisor's liveness probe runs on the machine it expects. The runner waits for forgectrl's supervisor to settle (motion verified, or the ladder's verdict) before and after every takeover.

Power-cycle before a campaign. forgetest takes a fresh-boot reference once per boot (/data/forgetest/boot-<boot_id>.json, taken only within the first ten minutes after boot, after the supervisor settles): the whole idle picture of this machine as the image boots it, the check on the fixed values, and the record a leftover is judged against. Take it after a power cycle, not a warm reboot - the machine's true fresh state is the powered-on one (the PIC's own lamp and sensor defaults, then forgectrl's start-up writes on top). A displaced head is jogged back along its own path by the kernel-measured X/Y delta (bounded to 100 mm; Z is never touched); beyond that the counters are reported and the run must be fixed. A run that legitimately re-zeroes the counters (cloud mode's connect) tells the runner so (ctx.counters_rezeroed()) and hands the head back itself.

The gate

scripts/release.sh <version> builds the release image, reads /etc/forgefirm-manifest.json out of the release rootfs and runs

scripts/acceptance-gate.py releases/v<version>/acceptance.json <manifest>

which requires: the artifact self-hash intact; authorized: true; the catalog in the tree identical to the artifact's; for every test a recorded PASS whose fingerprint equals the one recomputed from the release manifest; inherited results not core and newer than the invalidate epoch. Any problem dies before signing. FORGEFIRM_ACCEPTANCE_SKIP=1 bypasses the gate deliberately and prints a loud warning; it is never the default. The artifact is staged and attached to the GitHub release next to forgefirm.fw.

Because the dev image and the release image are built from the same tree in one bitbake invocation, their manifests share the same identity; a pin bumped after the campaign shows up as a fingerprint mismatch on exactly the tests that cover it.

Coverage currency rule

Every change is evaluated against the catalog, in addition to its unit tests:

1. does an existing test exercise the changed behavior - if not, add or extend one in the same change;
2. does that test's covers map name the files touched - if not, widen it in the same change.

A behavior change with no catalog consequence needs a sentence of justification in the commit message. Coverage gaps are defects: under the domain model an uncovered path lets an inherited PASS stay valid across a change that should have invalidated it.

The mechanical floor is the coverage lint,

python3 -m forgetest.coverage --manifest <manifest.json> [--enforce]

which lists every manifest path no test covers, minus the allowlist of non-behavioral paths in forgetest/forgetest/coverage.py (docs, CI, tests, licenses). CI (forgetest-ci.yml) runs it on a manifest generated from the recipe pins with scripts/manifest-from-tree.py (no Yocto build needed) and fails the job on any uncovered path. On the board, run it against /etc/forgefirm-manifest.json. The lint proves a file is fingerprinted; whether the test exercises the change is the change author's judgment (rule 1).

Bench diagnostics page

The same daemon serves #bench: the registry of the bench tools (scripts/bench, installed under /usr/share/forgetest/bench/), each with its safety class, argument form, and last run. The classes: dry (reads or dry motion, forgectrl stays up), takeover (forgectrl and the controller stopped for the run, the pulse device free, the same wrapper the takeover tests use), scope (a takeover whose result only means something with the named instrument on the bench), live (emission possible; the operator acknowledgment and the physical arm press). A tool runs as a subprocess with the output on the page and, on the board, the machine as GF_HOST=127.0.0.1, the panel token in GF_TOKEN, and its data files under /data/forgetest/bench/ (FORGETEST_BENCH_DATA); the same scripts run from a LAN host with GF_HOST set (scripts/bench/gfbench.py, scripts/bench/README.md). Every board-runnable tool is ported; the entries that stay unported are the CI harnesses of the null-sink controller and the factory .puls decoder, which do not run against the machine at all - they are listed so the catalog of what exists is complete. Bench runs are recorded in /data/forgetest/bench.jsonl and never enter a campaign.

Layout

forgetest/forgetest/         the package (stdlib only)
  manifest.py                manifest, globs, fingerprints, coverage report
  catalog.py                 @test registry, catalog hash
  campaign.py                the rules (pure functions)
  artifact.py                export + gate verification
  runner.py                  one run at a time, prompts, abort, takeover, queues
  baseline.py                the fresh-boot idle state around every run
  server.py / page.py        HTTP API + the page (forgectrl's access rules)
  bench.py / coverage.py     bench registry + subprocess runner; the lint
  suite/                     the catalog, one module per subsystem
forgetest/tests/             host unit tests (python3 -m unittest discover -s tests)
scripts/bench/               the bench tools (+ gfbench.py, the board/host helper)
scripts/acceptance-gate.py   the gate
scripts/manifest-from-tree.py  manifest from the recipe pins (CI, workstation)
releases/v<version>/         the committed artifacts