mirror of
https://github.com/openglow-org/forgefirm.git
synced 2026-10-15 17:19:55 -07:00
ScottW514
a9c794be34
The coverage lint failed CI on any behavioral path no catalog test covered, so every software change pulled a new or wider acceptance test in with it, though the catalog is for what needs the machine or an operator. A path whose behavior a host test in CI proves now has a row in HOST_TESTED (manifest.py): the component, a glob, the host test, and the CI workflow that runs it, the last two as "<repository>:<path>" - a file of a component in the manifest (its tests/ and .github/ files are listed there), or "forgefirm:" and a path in this repository (the null-sink harnesses, which grblhal-glowforge's workflow runs). - coverage_report counts a host-tested path as covered. Fingerprints are unchanged: a path no catalog test covers is in no fingerprint, so a change there makes no acceptance run necessary. - host_row_problems reports a row whose glob selects no behavioral file; a host test or workflow that does not exist (a component file missing from the manifest, a "forgefirm:" file missing from the checkout, the dev-only forgetest, no repository named); a workflow that should be readable and is not; one that does not check out the test's repository; and one with no step that runs the test. - The workflow is read at the pinned revision: read_source takes a component's blob from the bare repo manifest-from-tree.py fetched for its source URL (the checkout's .manifest-cache, or --sources), a "forgefirm:" file from the checkout. With no sources (an image) it is not read and the lint says so; with sources, a read that fails is a problem, never a pass. - workflow_steps reads the workflow's steps by indentation (uses, run, working-directory, with), comments dropped, whole lines and from " #"; a quoted scalar loses its quotes, a quote inside a command stays. checkout_path finds where the test's repository is checked out (its own: the checkout naming no repository; another: the checkout of <owner>/<name>, the name from the component's source URL). - workflow_runs looks at what a step's run commands execute, split into simple commands (;, &&, ||, |, newlines, then/do/else; a backslash continuation joined), with sudo and its flags, env, time, exec and variable assignments passed over, each path resolved against the step's working-directory and compared with the test's workspace path: the command itself (a script run directly, a C test's built binary <dir>/<stem>); the script an interpreter runs (python, bash, sh) or a for loop's list, by path or a glob matching the whole path segment by segment; unittest discover starting in the test's own directory with a pattern its name matches (test*.py unless -p); pytest given the test or its own directory, for a name it collects. An argument to anything else (echo, cat, a build target), a step's name, a comment, a trigger's paths filter, or a path that only ends like the test's does not count. - The lint prints the row problems and fails --enforce on them, as on an uncovered path or an empty covers entry; --json carries them as host_tested, with workflows_read (false when a workflow that should have been read was not). It reads HOST_TESTED when it runs. - The table starts empty: rows come with the tests that leave the catalog. Proof: host tests: a host-tested row takes its paths out of the report and moves no fingerprint; good rows report nothing; each kind of bad row is reported with its reason; a workflow that runs one harness and names another only in its paths filter, a comment line and an inline comment runs the first and not the second; workflow_runs on a built binary, sudo and a binary, a script, a harness under another checkout, pytest on a directory, a for-loop glob and unittest discover under a step's working-directory, and not under another one; and not on a paths entry, an inline comment, a glob or a discover that the path only ends like (tools/tests/..., scripts/bench/tests/...), a longer name, another file, a step's name, or a commented line in a block; not on echo with the stem or the binary's path, a build target, cat of a script, unittest discover from a directory above the test or with a pattern its name misses, or pytest on a directory above it; on discover with -p '*_test.py', pytest given the file, a script run directly and by bash, and sudo -E with assignments continued onto the next line; checkout_path for its own and another repository and none; repo_name from a source URL; a component's workflow read from a bare repo's blob, judged, not judged without the sources, and a blob that cannot be read reported; the command line with a workflow it cannot read reports workflows_read false and the read failure, and fails --enforce. The manifest tests (34) pass on Windows and under Linux; the forgetest host suite passes under Linux. Against the recipe pins and their fetched workflows: rows for forgectrl's tray_test, forgeext's holdkeep_test, sandbox_test (under sudo) and run_test.py (sudo, env and assignments over three lines), the laser_stream_test harness (grblhal-glowforge's checkout of forgefirm), gfhardware's test_machine_lid_button (a for-loop glob under its working-directory), gfutilities' test_frame (pytest on tests) and forgetest's own test_manifest (unittest discover under forgefirm/forgetest) pass; rows for forgectrl's fflog_e2e.sh, the bench script fire_test.py and kas/source-bundle.yml (only in forgetest-ci's paths filter) are refused. The lint on that manifest reports 0 uncovered paths and 0 host-tested row problems. Acceptance: none. forgetest is dev-only and in no fingerprint; the lint runs in CI, not in a campaign.