mirror of
https://github.com/openglow-org/forgefirm.git
synced 2026-10-15 17:19:55 -07:00
ScottW514
a9c794be34
The coverage lint failed CI on any behavioral path no catalog test covered, so every software change pulled a new or wider acceptance test in with it, though the catalog is for what needs the machine or an operator. A path whose behavior a host test in CI proves now has a row in HOST_TESTED (manifest.py): the component, a glob, the host test, and the CI workflow that runs it, the last two as "<repository>:<path>" - a file of a component in the manifest (its tests/ and .github/ files are listed there), or "forgefirm:" and a path in this repository (the null-sink harnesses, which grblhal-glowforge's workflow runs). - coverage_report counts a host-tested path as covered. Fingerprints are unchanged: a path no catalog test covers is in no fingerprint, so a change there makes no acceptance run necessary. - host_row_problems reports a row whose glob selects no behavioral file; a host test or workflow that does not exist (a component file missing from the manifest, a "forgefirm:" file missing from the checkout, the dev-only forgetest, no repository named); a workflow that should be readable and is not; one that does not check out the test's repository; and one with no step that runs the test. - The workflow is read at the pinned revision: read_source takes a component's blob from the bare repo manifest-from-tree.py fetched for its source URL (the checkout's .manifest-cache, or --sources), a "forgefirm:" file from the checkout. With no sources (an image) it is not read and the lint says so; with sources, a read that fails is a problem, never a pass. - workflow_steps reads the workflow's steps by indentation (uses, run, working-directory, with), comments dropped, whole lines and from " #"; a quoted scalar loses its quotes, a quote inside a command stays. checkout_path finds where the test's repository is checked out (its own: the checkout naming no repository; another: the checkout of <owner>/<name>, the name from the component's source URL). - workflow_runs looks at what a step's run commands execute, split into simple commands (;, &&, ||, |, newlines, then/do/else; a backslash continuation joined), with sudo and its flags, env, time, exec and variable assignments passed over, each path resolved against the step's working-directory and compared with the test's workspace path: the command itself (a script run directly, a C test's built binary <dir>/<stem>); the script an interpreter runs (python, bash, sh) or a for loop's list, by path or a glob matching the whole path segment by segment; unittest discover starting in the test's own directory with a pattern its name matches (test*.py unless -p); pytest given the test or its own directory, for a name it collects. An argument to anything else (echo, cat, a build target), a step's name, a comment, a trigger's paths filter, or a path that only ends like the test's does not count. - The lint prints the row problems and fails --enforce on them, as on an uncovered path or an empty covers entry; --json carries them as host_tested, with workflows_read (false when a workflow that should have been read was not). It reads HOST_TESTED when it runs. - The table starts empty: rows come with the tests that leave the catalog. Proof: host tests: a host-tested row takes its paths out of the report and moves no fingerprint; good rows report nothing; each kind of bad row is reported with its reason; a workflow that runs one harness and names another only in its paths filter, a comment line and an inline comment runs the first and not the second; workflow_runs on a built binary, sudo and a binary, a script, a harness under another checkout, pytest on a directory, a for-loop glob and unittest discover under a step's working-directory, and not under another one; and not on a paths entry, an inline comment, a glob or a discover that the path only ends like (tools/tests/..., scripts/bench/tests/...), a longer name, another file, a step's name, or a commented line in a block; not on echo with the stem or the binary's path, a build target, cat of a script, unittest discover from a directory above the test or with a pattern its name misses, or pytest on a directory above it; on discover with -p '*_test.py', pytest given the file, a script run directly and by bash, and sudo -E with assignments continued onto the next line; checkout_path for its own and another repository and none; repo_name from a source URL; a component's workflow read from a bare repo's blob, judged, not judged without the sources, and a blob that cannot be read reported; the command line with a workflow it cannot read reports workflows_read false and the read failure, and fails --enforce. The manifest tests (34) pass on Windows and under Linux; the forgetest host suite passes under Linux. Against the recipe pins and their fetched workflows: rows for forgectrl's tray_test, forgeext's holdkeep_test, sandbox_test (under sudo) and run_test.py (sudo, env and assignments over three lines), the laser_stream_test harness (grblhal-glowforge's checkout of forgefirm), gfhardware's test_machine_lid_button (a for-loop glob under its working-directory), gfutilities' test_frame (pytest on tests) and forgetest's own test_manifest (unittest discover under forgefirm/forgetest) pass; rows for forgectrl's fflog_e2e.sh, the bench script fire_test.py and kas/source-bundle.yml (only in forgetest-ci's paths filter) are refused. The lint on that manifest reports 0 uncovered paths and 0 host-tested row problems. Acceptance: none. forgetest is dev-only and in no fingerprint; the lint runs in CI, not in a campaign.
93 lines
3.5 KiB
YAML
93 lines
3.5 KiB
YAML
# forgetest: the release acceptance tool's host-side proof.
|
|
#
|
|
# - unit tests: campaign rules, fingerprints, artifact build/verify (the
|
|
# release gate's decision, including the negative fixtures), runner +
|
|
# HTTP API end to end with a fake catalog and a fake bench tool, the
|
|
# suites replayed on the machine's own log lines, and the check that
|
|
# every log phrase the cloud suite greps for is one the pinned cloud
|
|
# app can log (it reads the app sources from the manifest cache)
|
|
# - the source bundle of a release: which recipe of the image needs
|
|
# source, which archive answers for it, and the refusal that stops a
|
|
# release with source missing
|
|
# - shared UI files: theme.css and the vendored Bootstrap the page
|
|
# carries are byte-identical to forgectrl's at its pinned revision
|
|
# - coverage lint: every source path of every component pinned by the
|
|
# recipes must be selected by some catalog test's coverage globs or by a
|
|
# host-tested row naming the host test that proves it (the tree manifest
|
|
# is generated from the pins with git - no Yocto build); enforced: an
|
|
# uncovered path fails the job (https://docs.forgefirm.org/developers/testing/).
|
|
|
|
name: forgetest-ci
|
|
|
|
on:
|
|
push:
|
|
branches: [master]
|
|
paths:
|
|
- 'forgetest/**'
|
|
- 'scripts/acceptance-gate.py'
|
|
- 'scripts/manifest-from-tree.py'
|
|
- 'scripts/source-bundle.py'
|
|
- 'kas/source-bundle.yml'
|
|
- 'meta-forgefirm/recipes-forgefirm/**'
|
|
- '.github/workflows/forgetest-ci.yml'
|
|
pull_request:
|
|
workflow_dispatch:
|
|
|
|
jobs:
|
|
test:
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 20
|
|
steps:
|
|
- name: Checkout forgefirm
|
|
uses: actions/checkout@v4
|
|
with:
|
|
path: forgefirm
|
|
|
|
- name: Checkout meta-openglow (sibling)
|
|
uses: actions/checkout@v4
|
|
with:
|
|
repository: openglow-org/meta-openglow
|
|
ref: scarthgap
|
|
path: meta-openglow
|
|
|
|
- name: Set up Python
|
|
uses: actions/setup-python@v5
|
|
with:
|
|
python-version: '3.12'
|
|
|
|
- name: Tree manifest from the recipe pins (also fetches the pinned app sources the needle check reads)
|
|
working-directory: forgefirm
|
|
run: python scripts/manifest-from-tree.py --out tree-manifest.json
|
|
|
|
- name: Unit tests
|
|
working-directory: forgefirm/forgetest
|
|
# -B: no bytecode caches under the package directory (the recipe's fetch
|
|
# checksum would otherwise move without a source change)
|
|
run: python -B -m unittest discover -s tests -v
|
|
|
|
- name: No undefined names (the live drills never execute on the host)
|
|
working-directory: forgefirm/forgetest
|
|
run: |
|
|
pip install pyflakes
|
|
if python -B -m pyflakes forgetest tests | grep -E "undefined name"; then exit 1; fi
|
|
|
|
- name: Shared UI files match forgectrl at its pinned revision
|
|
working-directory: forgefirm
|
|
run: python scripts/check-ui-vendor.py
|
|
|
|
- name: Coverage lint (enforced)
|
|
working-directory: forgefirm/forgetest
|
|
run: python -B -m forgetest.coverage --manifest ../tree-manifest.json --enforce | tee "$GITHUB_STEP_SUMMARY"
|
|
|
|
- name: Gate self-check on the tree manifest
|
|
working-directory: forgefirm
|
|
run: |
|
|
# The gate must load the catalog and refuse a missing/unauthorized
|
|
# artifact cleanly (exit 1/2, never a traceback).
|
|
set +e
|
|
echo '{}' > empty.json
|
|
python scripts/acceptance-gate.py empty.json tree-manifest.json --quiet
|
|
rc=$?
|
|
set -e
|
|
test "$rc" -eq 1
|