vh/forgefirm
1
0
Fork 0
mirror of https://github.com/openglow-org/forgefirm.git synced 2026-10-19 02:51:42 -07:00
Code Issues Packages Projects Releases Wiki Activity

forgetest: the lint counts a step as run only when every job its job needs runs

Browse Source 
The coverage lint judges a HOST_TESTED row by a step of its workflow
that runs on every run of the workflow. It read the `if:` and the
`continue-on-error:` of a step and of its job, and not the job's
`needs:`. GitHub skips a job whose needed job did not run (an `if:`
that held it back, or a failure), unless the job says `always()`, so
a test step in a job that needs a conditional job runs on not every
push, and counted all the same: a host test behind a conditional
build keeps its row green and its paths covered. No workflow a row
names uses `needs:`.

workflow_steps now reads each job's `needs:` (_needs: a flow list,
one scalar, or a block list), keeps the jobs by name, and resolves
them when the workflow is read (_needs_run): a job's needs are met
on every run when every job it names runs on every run, that is,
carries no `if:` other than the literal true and has its own needs
met, transitively. A job the workflow does not have, a cycle, and a
`needs:` in a form the lint does not read each fail closed. A needed
job's `continue-on-error:` does not bear: its failure fails neither
the run nor the jobs after it. The job's dict carries the result as
"needs_run", and _on_every_run counts no step of a job whose needs
are not met, so workflow_runs and checkout_path refuse it as they
refuse a conditional step. The reason stays "does not run it". The
lint evaluates no condition, so a dependent job under `always()`
counts as no job, as a step under `always()` counts as no step. The
HOST_TESTED comment and the docstrings say what the lint reads.

Proof: forgetest-ci "Unit tests" (`python -B -m unittest discover -s
tests -v` in forgetest): test_manifest.py runs 40 tests, OK, on
Ubuntu 24.04 with python 3.12, in a private PID namespace. The new
case test_a_job_runs_only_when_every_job_it_needs_runs: a job that
needs an unconditional job runs its test, with the `needs:` as a
scalar, a flow list, a quoted flow list and a block list; the same
four with the needed job under `if: false` or `if: github.event_name
== 'push'` do not; a needed job under `continue-on-error: true`
runs it; a chain of three runs it, and does not with the first job
under `if: false`; a needed job the workflow does not have, a cycle
of two, a self-need and a `needs:` holding a mapping each do not;
`if: always()` on the dependent job, and on the middle job of the
chain, does not. test_the_workflow_has_to_run_the_host_test gains a
workflow whose test job needs a job under `if: false`, listed as
"does not run it". Control: the parent, e1d1c1b, with the new test
file fails both cases (failures=2). The coverage lint on the tree
manifest from the recipe pins, with meta-openglow at 1e163bd on
scarthgap, reports 0 uncovered paths, 0 empty entries, 0 host-tested
row problems, 43 tests. pyflakes reports no undefined name.

Acceptance: none. forgetest is dev-only and in no fingerprint; the
lint runs in CI, not in a campaign, and tests/ are in no fingerprint.
This commit is contained in:
ScottW514
2026-10-18 13:40:54 -04:00
parent e1d1c1b84e
commit 9c97f05db9
2 changed files with 110 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files
forgetest/tests/test_manifest.py
+39
View File
@@ -254,6 +254,16 @@ class CoverageReportTests(unittest.TestCase):
self.assertEqual(m.host_row_problems(man, [manual, soft], root=root, sources=None),
[manual[:3] + (manual[3] + ": the lint reads no push or pull_request in its own on:",),
soft[:3] + (soft[3] + " does not run it",)])
# a step in a job that needs a conditional job does not run it (the checkout in a job of
# its own, as the lint reads a checkout for the whole workflow)
root = self._root({"scripts/bench/stream_test.py": "",
".github/workflows/needs.yml": "on: [push]\njobs:\n lint:\n steps:\n"
" - uses: actions/checkout@v4\n build:\n if: false\n steps:\n"
" - run: make\n test:\n needs: build\n steps:\n"
" - run: python3 scripts/bench/stream_test.py\n"})
needs = runs[:3] + ("forgefirm:.github/workflows/needs.yml",)
self.assertEqual(m.host_row_problems(man, [needs], root=root, sources=None),
[needs[:3] + (needs[3] + " does not run it",)])
def _runs(self, yml, path):
return m.workflow_runs(m.workflow_steps(yml), path)
@@ -396,6 +406,35 @@ class CoverageReportTests(unittest.TestCase):
self.assertEqual(m.checkout_path(m.workflow_steps(yml % ""), "x", True), "own")
self.assertIsNone(m.checkout_path(m.workflow_steps(yml % " if: false\n"), "x", True))
def test_a_job_runs_only_when_every_job_it_needs_runs(self):
test = "tests/tray_test.c"
# GitHub skips a job whose needed job did not run: a job that needs a conditional job runs
# on not every run, however the needs: is written
yml = "jobs:\n build:\n%s steps:\n - run: make\n test:\n%s steps:\n - run: ./build/tray_test\n"
for needs in (" needs: build\n", " needs: [build]\n", " needs: ['build']\n",
" needs:\n - build\n"):
self.assertTrue(self._runs(yml % ("", needs), test), needs)
self.assertFalse(self._runs(yml % (" if: false\n", needs), test), needs)
self.assertFalse(self._runs(yml % (" if: github.event_name == 'push'\n", needs), test), needs)
# the needed job's continue-on-error: does not bear: its failure fails neither the run nor
# the jobs after it
self.assertTrue(self._runs(yml % (" continue-on-error: true\n", " needs: build\n"), test))
# transitively; a job the workflow does not have, a cycle, and a needs: the lint does not
# read each fail closed
chain = "jobs:\n a:\n%s steps:\n - run: make\n b:\n needs: a\n steps:\n - run: make\n" \
" c:\n needs: [b]\n steps:\n - run: ./build/tray_test\n"
self.assertTrue(self._runs(chain % "", test))
self.assertFalse(self._runs(chain % " if: false\n", test))
self.assertFalse(self._runs(yml % ("", " needs: lint\n"), test))
self.assertFalse(self._runs(yml % (" needs: test\n", " needs: build\n"), test))
self.assertFalse(self._runs(yml % ("", " needs: test\n"), test))
self.assertFalse(self._runs(yml % ("", " needs:\n build: true\n"), test))
# a job under always() runs on every run, and the lint evaluates no condition: it counts as
# no job for its own steps and for the jobs after it
self.assertFalse(self._runs(yml % (" if: false\n", " needs: build\n if: always()\n"), test))
self.assertFalse(self._runs((chain % " if: false\n").replace(
" b:\n needs: a\n", " b:\n needs: a\n if: ${{ always() }}\n"), test))
def test_the_events_a_workflow_runs_on(self):
self.assertEqual(m.workflow_triggers("on:\n push:\n branches: [main]\n pull_request:\njobs:\n t:\n"),
{"push", "pull_request"})