mirror of
https://github.com/openglow-org/forgefirm.git
synced 2026-10-15 17:19:55 -07:00
ScottW514
a9c794be34
The coverage lint failed CI on any behavioral path no catalog test covered, so every software change pulled a new or wider acceptance test in with it, though the catalog is for what needs the machine or an operator. A path whose behavior a host test in CI proves now has a row in HOST_TESTED (manifest.py): the component, a glob, the host test, and the CI workflow that runs it, the last two as "<repository>:<path>" - a file of a component in the manifest (its tests/ and .github/ files are listed there), or "forgefirm:" and a path in this repository (the null-sink harnesses, which grblhal-glowforge's workflow runs). - coverage_report counts a host-tested path as covered. Fingerprints are unchanged: a path no catalog test covers is in no fingerprint, so a change there makes no acceptance run necessary. - host_row_problems reports a row whose glob selects no behavioral file; a host test or workflow that does not exist (a component file missing from the manifest, a "forgefirm:" file missing from the checkout, the dev-only forgetest, no repository named); a workflow that should be readable and is not; one that does not check out the test's repository; and one with no step that runs the test. - The workflow is read at the pinned revision: read_source takes a component's blob from the bare repo manifest-from-tree.py fetched for its source URL (the checkout's .manifest-cache, or --sources), a "forgefirm:" file from the checkout. With no sources (an image) it is not read and the lint says so; with sources, a read that fails is a problem, never a pass. - workflow_steps reads the workflow's steps by indentation (uses, run, working-directory, with), comments dropped, whole lines and from " #"; a quoted scalar loses its quotes, a quote inside a command stays. checkout_path finds where the test's repository is checked out (its own: the checkout naming no repository; another: the checkout of <owner>/<name>, the name from the component's source URL). - workflow_runs looks at what a step's run commands execute, split into simple commands (;, &&, ||, |, newlines, then/do/else; a backslash continuation joined), with sudo and its flags, env, time, exec and variable assignments passed over, each path resolved against the step's working-directory and compared with the test's workspace path: the command itself (a script run directly, a C test's built binary <dir>/<stem>); the script an interpreter runs (python, bash, sh) or a for loop's list, by path or a glob matching the whole path segment by segment; unittest discover starting in the test's own directory with a pattern its name matches (test*.py unless -p); pytest given the test or its own directory, for a name it collects. An argument to anything else (echo, cat, a build target), a step's name, a comment, a trigger's paths filter, or a path that only ends like the test's does not count. - The lint prints the row problems and fails --enforce on them, as on an uncovered path or an empty covers entry; --json carries them as host_tested, with workflows_read (false when a workflow that should have been read was not). It reads HOST_TESTED when it runs. - The table starts empty: rows come with the tests that leave the catalog. Proof: host tests: a host-tested row takes its paths out of the report and moves no fingerprint; good rows report nothing; each kind of bad row is reported with its reason; a workflow that runs one harness and names another only in its paths filter, a comment line and an inline comment runs the first and not the second; workflow_runs on a built binary, sudo and a binary, a script, a harness under another checkout, pytest on a directory, a for-loop glob and unittest discover under a step's working-directory, and not under another one; and not on a paths entry, an inline comment, a glob or a discover that the path only ends like (tools/tests/..., scripts/bench/tests/...), a longer name, another file, a step's name, or a commented line in a block; not on echo with the stem or the binary's path, a build target, cat of a script, unittest discover from a directory above the test or with a pattern its name misses, or pytest on a directory above it; on discover with -p '*_test.py', pytest given the file, a script run directly and by bash, and sudo -E with assignments continued onto the next line; checkout_path for its own and another repository and none; repo_name from a source URL; a component's workflow read from a bare repo's blob, judged, not judged without the sources, and a blob that cannot be read reported; the command line with a workflow it cannot read reports workflows_read false and the read failure, and fails --enforce. The manifest tests (34) pass on Windows and under Linux; the forgetest host suite passes under Linux. Against the recipe pins and their fetched workflows: rows for forgectrl's tray_test, forgeext's holdkeep_test, sandbox_test (under sudo) and run_test.py (sudo, env and assignments over three lines), the laser_stream_test harness (grblhal-glowforge's checkout of forgefirm), gfhardware's test_machine_lid_button (a for-loop glob under its working-directory), gfutilities' test_frame (pytest on tests) and forgetest's own test_manifest (unittest discover under forgefirm/forgetest) pass; rows for forgectrl's fflog_e2e.sh, the bench script fire_test.py and kas/source-bundle.yml (only in forgetest-ci's paths filter) are refused. The lint on that manifest reports 0 uncovered paths and 0 host-tested row problems. Acceptance: none. forgetest is dev-only and in no fingerprint; the lint runs in CI, not in a campaign.
104 lines
4.5 KiB
Python
104 lines
4.5 KiB
Python
# Copyright 2026 514 LLC d/b/a OpenGlow
|
|
# Written by Scott Wiederhold
|
|
# https://community.openglow.org
|
|
# SPDX-License-Identifier: MIT
|
|
|
|
"""Coverage lint: every source path in the manifest must be selected by
|
|
some test's coverage globs or by a host-tested row (manifest.HOST_TESTED),
|
|
except the allowlisted non-behavioral paths; every coverage glob and every
|
|
host-tested glob must select at least one path of its component; and every
|
|
host-tested row must name a host test that exists and a CI workflow that
|
|
runs it.
|
|
|
|
Why a hard rule: under the domain model an uncovered file is worse than an
|
|
untested one - a change there leaves every inherited PASS valid when it
|
|
should have invalidated them. The lint is the floor (the file is
|
|
fingerprinted by at least one test, or a named host test that CI runs
|
|
answers for it); whether that test exercises the change stays with the
|
|
change author. An entry that selects nothing is the same defect from the
|
|
other side: the test's fingerprint ignores the file it named.
|
|
|
|
Whether a workflow runs its host test is read from the workflow's text at
|
|
the pinned revision, in the sources scripts/manifest-from-tree.py fetched
|
|
(--sources, default the checkout's .manifest-cache). Without them (on an
|
|
image) that part is not checked, and the report says so.
|
|
|
|
python3 -m forgetest.coverage [--manifest PATH] [--sources DIR] [--enforce] [--json]
|
|
|
|
Exit 0 when nothing is uncovered (or when only reporting), 1 under
|
|
--enforce with uncovered paths, 2 on a usage or load error.
|
|
"""
|
|
import argparse
|
|
import json
|
|
import os
|
|
import sys
|
|
|
|
from . import catalog as _catalog
|
|
from . import manifest as _manifest
|
|
|
|
# Non-behavioral paths need no acceptance coverage, and no fingerprint
|
|
# carries them: the one list, in the manifest module.
|
|
ALLOW = _manifest.NON_BEHAVIORAL
|
|
|
|
|
|
def run(manifest, tests, allow=ALLOW):
|
|
return _manifest.coverage_report(manifest, tests, allow, host=_manifest.HOST_TESTED)
|
|
|
|
|
|
def empty(manifest, tests):
|
|
return _manifest.empty_covers(manifest, tests)
|
|
|
|
|
|
def host_problems(manifest, sources=_manifest.SOURCES):
|
|
return _manifest.host_row_problems(manifest, host=_manifest.HOST_TESTED, sources=sources)
|
|
|
|
|
|
def main(argv=None):
|
|
ap = argparse.ArgumentParser(description="forgetest coverage lint")
|
|
ap.add_argument("--manifest", default=None, help="manifest JSON (default: the running image's)")
|
|
ap.add_argument("--sources", default=_manifest.SOURCES,
|
|
help="the pinned sources manifest-from-tree.py fetched (default: the checkout's .manifest-cache)")
|
|
ap.add_argument("--enforce", action="store_true", help="exit 1 when any path is uncovered")
|
|
ap.add_argument("--json", action="store_true", help="machine-readable report")
|
|
args = ap.parse_args(argv)
|
|
try:
|
|
manifest = _manifest.Manifest.load(args.manifest)
|
|
except (OSError, ValueError) as e:
|
|
print("cannot load manifest: %s" % e, file=sys.stderr)
|
|
return 2
|
|
registry = _catalog.load_suite()
|
|
tests = _catalog.all_tests(registry)
|
|
report = run(manifest, tests)
|
|
total = sum(len(v) for v in report.values())
|
|
hollow = empty(manifest, tests)
|
|
sources = args.sources if args.sources and os.path.isdir(args.sources) else None
|
|
rows = host_problems(manifest, sources)
|
|
if args.json:
|
|
print(json.dumps({"uncovered": report, "total": total, "tests": len(tests),
|
|
"empty": [list(e) for e in hollow],
|
|
"host_tested": [list(r) for r in rows],
|
|
"workflows_read": sources is not None and not any("cannot be read" in r[3] for r in rows)},
|
|
indent=1, sort_keys=True))
|
|
else:
|
|
if _manifest.HOST_TESTED and sources is None:
|
|
print("host-tested: no sources, so whether a workflow runs each host test is not checked")
|
|
for comp in sorted(report):
|
|
print("%s: %d uncovered path(s)" % (comp, len(report[comp])))
|
|
for p in report[comp]:
|
|
print(" %s" % p)
|
|
for tid, comp, pat in hollow:
|
|
print("%s: covers (%s, %s) selects nothing" % (tid, comp, pat))
|
|
for comp, pat, ref, why in rows:
|
|
print("host-tested (%s, %s) by %s: %s" % (comp, pat, ref, why))
|
|
print("coverage: %d uncovered path(s) across %d component(s), %d empty entr%s, "
|
|
"%d host-tested row problem(s), %d tests"
|
|
% (total, len(report), len(hollow), "y" if len(hollow) == 1 else "ies", len(rows),
|
|
len(tests)))
|
|
if args.enforce and (total or hollow or rows):
|
|
return 1
|
|
return 0
|
|
|
|
|
|
if __name__ == "__main__":
|
|
sys.exit(main())
|