mirror of
https://github.com/openglow-org/forgefirm.git
synced 2026-10-19 02:51:42 -07:00
forgetest: the shared judges and a test's own imports reach the hash the gate reads
The sibling walk read the top-level statements of a module alone. Five
tests import their judging helpers inside their own functions, so that
no other test of the module moves with them: cloud.mode-switch
(homeoff.py: judge_whole_motions, camera_home_return),
cooling.flow-under-load and cooling.fail-tier-stop (laser.py's sample
and run_and_sample, motion.py's log readers), forgectrl.auth (setup.py's
request and decode) and motion.release (setup_dark.py's run_check).
None of those helpers was in the hash of its test, and homeoff.py was
in no hash at all: an edit to judge_whole_motions moved 0 of 43 tests,
and an edit to laser.sample moved the three laser.* tests and neither
cooling test, so an inherited PASS survived a change to its own judge.
The guard that every package module a suite module imports is a listed
judge read the top level alone too, and no unit test held the
production path: Test.source_sha, which the fingerprint, the campaign
and the gate read, could drop the shared judges with CI green.
implementation_sha now adds to the hash of a test the suite modules its
own @test function imports, at any depth, and what they import in turn;
no other test of the module takes them. The shared text of a module
gives the modules it imports at any depth too: a helper that imports
inside its body brings that module into every test of the module, as
the helper itself does. _test_defs finds the @test functions of a
module for module_parts and for both walks. The guard walks the whole
tree of each suite module, so a package import inside a function or a
block is held to the list as well. test_a_shared_judge_edit_moves_every_test
holds the production path on a suite test: its source_sha equals
implementation_sha with SHARED_JUDGES and differs from it with none.
Proof: forgetest-ci "Unit tests" (`python -B -m unittest discover -s
tests -v` in forgetest): test_manifest.py runs 39 tests, OK, on Windows
with python 3.14 and on Ubuntu 24.04 with python 3.12, with
test_a_sibling_imported_inside_a_function_moves_the_tests_that_run_it
(a judge imported in the body of one test moves that test and not the
other; imported in the body of a helper, it moves both), the
source_sha assertions, and the guard on the whole tree; test_artifact.py,
test_campaign.py, test_redact.py and test_responsiveness.py pass.
Negative controls on scratch copies: implementation_sha's default
taking no judges fails the source_sha assertion, and so does
Test.source_sha passing judges=(); the new case fails against
catalog.py as it was, and against a shared-text walk of top-level
statements alone; a `from .. import server` inside a function of
suite/forgectrl.py, or in a try block at its top level, fails the
guard, and the function-local one passes the guard that reads the top
level alone. On the real catalog: an edit to judge_whole_motions moves
cloud.mode-switch alone (0 tests before); an edit to laser.sample moves
the three laser.* tests and both cooling tests (the three alone
before); an edit to run_check moves the six setup.* tests and
motion.release (the six alone before); an edit to setup.request moves
forgectrl.auth and motion.release (motion.release through setup_dark.py,
which imports setup.py) beside the 8 it moved before; an edit to
hw.sysfs_int
moves 43 of 43, as before. The coverage lint on the tree manifest
from the recipe pins reports 0 uncovered paths, 0 host-tested row
problems, 43 tests. pyflakes reports no undefined name.
Acceptance: cloud.mode-switch, cooling.flow-under-load,
cooling.fail-tier-stop, forgectrl.auth and motion.release: the
implementation hash of each moves once, its own imports joining it; the
other 38 do not move (source_sha compared before and after). Against
the campaign this series already needs, that adds no run: 850498f moved
all 43 once. tests/ are in no fingerprint.
This commit is contained in:
@@ -485,8 +485,9 @@ class CatalogTests(unittest.TestCase):
|
||||
def test_the_shared_judges_exist_and_take_every_package_import_of_the_suite(self):
|
||||
# The list is kept by hand: the Context reaches runner.py, hw.py
|
||||
# and baseline.py with no import to find. So every package module
|
||||
# a suite module does import has to be on it (or be the registry,
|
||||
# which judges nothing), or that judge would be in no fingerprint.
|
||||
# a suite module does import, at its top level or inside a function
|
||||
# or a block, has to be on it (or be the registry, which judges
|
||||
# nothing), or that judge would be in no fingerprint.
|
||||
import ast
|
||||
import glob
|
||||
paths = catalog.shared_judge_paths()
|
||||
@@ -497,7 +498,7 @@ class CatalogTests(unittest.TestCase):
|
||||
for mod in sorted(glob.glob(os.path.join(catalog.suite_dir(), "*.py"))):
|
||||
with open(mod, encoding="utf-8") as f:
|
||||
tree = ast.parse(f.read())
|
||||
for node in tree.body:
|
||||
for node in ast.walk(tree):
|
||||
if isinstance(node, ast.ImportFrom) and node.level == 2:
|
||||
for name in ([node.module] if node.module else [a.name for a in node.names]):
|
||||
self.assertIn(name + ".py", allowed,
|
||||
@@ -614,6 +615,41 @@ def two(ctx):
|
||||
with open(other, "w", newline="\n") as f:
|
||||
f.write("def serve():\n return 1\n")
|
||||
self.assertEqual((a2, b2), shas())
|
||||
# Test.source_sha, which the fingerprint, the campaign and the gate
|
||||
# read, takes the shared judges itself: a suite test's hash is the
|
||||
# one with SHARED_JUDGES folded in, not the one without them.
|
||||
import inspect
|
||||
t = catalog.load_suite()["image.health"]
|
||||
src = inspect.getsourcefile(t.fn)
|
||||
self.assertEqual(t.source_sha, catalog.implementation_sha(src, t.id, judges=catalog.shared_judge_paths()))
|
||||
self.assertNotEqual(t.source_sha, catalog.implementation_sha(src, t.id, judges=[]))
|
||||
|
||||
def test_a_sibling_imported_inside_a_function_moves_the_tests_that_run_it(self):
|
||||
# A test that imports a judge inside its own function carries the
|
||||
# judge in its hash, and no other test of its module does; a
|
||||
# helper of the module that imports one inside its body carries it
|
||||
# into every test of the module.
|
||||
sib = os.path.join(self.tmp, "judge.py")
|
||||
|
||||
def write_judge(text):
|
||||
with open(sib, "w", newline="\n") as f:
|
||||
f.write(text)
|
||||
catalog.forget(sib)
|
||||
write_judge("def judge(x):\n return x > 1\n")
|
||||
self.write(self.MODULE.replace("def two(ctx):\n",
|
||||
"def two(ctx):\n from .judge import judge\n judge(2)\n"))
|
||||
a1, b1 = self.shas()
|
||||
write_judge("def judge(x):\n return x > 2\n")
|
||||
a2, b2 = self.shas()
|
||||
self.assertEqual(a1, a2)
|
||||
self.assertNotEqual(b1, b2)
|
||||
self.write(self.MODULE.replace(" return x + 1\n",
|
||||
" from .judge import judge\n return judge(x)\n"))
|
||||
a3, b3 = self.shas()
|
||||
write_judge("def judge(x):\n return x > 3\n")
|
||||
a4, b4 = self.shas()
|
||||
self.assertNotEqual(a3, a4)
|
||||
self.assertNotEqual(b3, b4)
|
||||
|
||||
def test_line_endings_do_not_count(self):
|
||||
a1, b1 = self.shas()
|
||||
|
||||
Reference in New Issue
Block a user