forgetest: the shared judges join every test's implementation hash

Every catalog test judges the machine through package modules it never
imports: the Context a test is handed and the takeover (runner.py), the
sysfs, LED, switch and HTTP readers behind it (hw.py), and the
fresh-boot idle state judged around every run (baseline.py). The cloud
and extension tests print the dark job of puls.py, which the suite
imports at level 2, and the sibling walk follows level-1 imports only.
So none of the four was in any fingerprint: a weakening of sysfs_int,
button_lit, Context.check, wait_for or position_quantized moved no
hash, and an inherited PASS of kernel.latch-locked-idle,
laser.armed-kill or any takeover test survived it unseen by the gate.

catalog.py names them in SHARED_JUDGES, with the criterion in the
comment above it: a module every test runs through without importing
it, or the dark job the cloud tests print. The rest of the package (the
server, the page, the campaign rules, the artifact, the manifest, the
registry, the bench actuator) judges nothing about the machine and
stays out. shared_judge_paths() resolves the four next to the module,
and implementation_sha folds their text after the module's shared text
and its imported siblings. A listed file that is missing raises, never
a hash computed without it. _direct_siblings stays level-1 only, and
the tool version does not move: the gate judges by source_sha and the
fingerprint.

Every catalog test's implementation hash moves once with this change,
so every recorded PASS is stale and the next campaign is a full one, 43
tests. The catalog hash does not move, so an open campaign stays open
with every test required. From here on, an edit to runner.py, hw.py,
baseline.py or puls.py moves every test.

Proof: forgetest-ci "Unit tests" (`python -B -m unittest discover -s
tests -v` in forgetest): test_manifest.py runs 38 tests, OK, with
test_a_shared_judge_edit_moves_every_test (an edit to a judge moves
both tests of a module, a package file outside the list moves neither)
and test_the_shared_judges_exist_and_take_every_package_import_of_the_suite
(the four files exist next to catalog.py, and every level-2 import of
a suite module is a listed judge or the registry). Negative controls
on scratch copies: `judged = []` in implementation_sha fails the first
case alone; "runner.py" dropped from SHARED_JUDGES fails the guard
alone, naming homeoff.py's import of runner. On the real catalog the
source_sha of 43 of 43 tests differs from HEAD's; with hw.sysfs_int
weakened to return the latch bit on an unparsable read, 0 of 43 move at
HEAD and 43 of 43 move with this change. The coverage lint on the tree
manifest from the recipe pins reports 0 uncovered paths, 0 host-tested
row problems, 43 tests. pyflakes reports no undefined name.

Acceptance: every catalog test. The implementation hash of all 43 moves
once, so the next campaign runs the whole catalog.

forgetest/forgetest/catalog.py

@@ -69,11 +69,13 @@ class Test:
     def source_sha(self):
         """The hash of the test's own implementation: its function (the
         decorator included) together with the code its module shares
-        among its tests - everything outside the @test functions. Part
-        of the fingerprint: a change inside one test's body invalidates
-        that test's earlier passes and no other; a change to a helper
-        invalidates the tests of that module. A test the module does not
-        define in the @test form hashes its whole file."""
+        among its tests - everything outside the @test functions - and
+        the shared judges (SHARED_JUDGES). Part of the fingerprint: a
+        change inside one test's body invalidates that test's earlier
+        passes and no other; a change to a helper invalidates the tests
+        of that module; a change to a shared judge invalidates every
+        test. A test the module does not define in the @test form hashes
+        its whole file."""
         if self._source_sha is None:
             # One filler at a time: a second poll that arrives while the
             # first computes waits for it rather than repeating the work.
@@ -144,6 +146,26 @@ def source_file_sha(path):
     return hashlib.sha256(data).hexdigest()
 
 
+# The package modules every test runs through without importing them:
+# the Context a test is handed and the takeover (runner.py), the sysfs,
+# LED, switch and HTTP readers behind it (hw.py), the fresh-boot idle
+# state judged around every run (baseline.py), and the dark job the
+# cloud and extension tests print (puls.py). Their text is part of every
+# judge, so it joins every implementation hash: a change to one of them
+# moves every test. The rest of the package (the server, the page, the
+# campaign rules, the artifact, the manifest, this registry, the bench
+# actuator) judges nothing about the machine and stays out.
+SHARED_JUDGES = ("baseline.py", "hw.py", "puls.py", "runner.py")
+
+
+def shared_judge_paths():
+    """The SHARED_JUDGES files, next to this module. Every one has to be
+    there: a judge missing from the package is an error, never a hash
+    computed without it."""
+    here = os.path.dirname(os.path.abspath(__file__))
+    return [os.path.join(here, name) for name in SHARED_JUDGES]
+
+
 # The files never change under a run, so each is read and parsed once:
 # path -> (text, tree or None), path -> (shared sha, {test id: own sha}),
 # path -> the sibling modules it imports itself.
@@ -241,15 +263,19 @@ def _direct_siblings(path):
         return hit
 
 
-def implementation_sha(path, test_id):
+def implementation_sha(path, test_id, judges=None):
     """The implementation hash of one test (see Test.source_sha): the
-    test's own text, the shared text of its module, and the shared text
-    of every sibling suite module the module imports."""
+    test's own text, the shared text of its module, the shared text of
+    every sibling suite module the module imports, and the text of the
+    shared judges. `judges` names files to take as the judges in place
+    of SHARED_JUDGES (the unit tests edit their own)."""
     shared, own = module_parts(path)
     if test_id not in own:
         return source_file_sha(path)
     helpers = [module_parts(p)[0] for p in sorted(sibling_imports(path))]
-    return _manifest.sha256_text("%s:%s" % (":".join([shared] + helpers), own[test_id]))
+    paths = shared_judge_paths() if judges is None else list(judges)
+    judged = [module_parts(p)[0] for p in paths]
+    return _manifest.sha256_text("%s:%s" % (":".join([shared] + helpers + judged), own[test_id]))
 
 
 def test(id, *, title, subsystem, kind="auto", hardware="api", mode=None, covers=(),

forgetest/tests/test_manifest.py

@@ -479,6 +479,27 @@ class CatalogTests(unittest.TestCase):
         for t in catalog.all_tests(reg):
             self.assertEqual(len(t.source_sha), 64)
 
+    def test_the_shared_judges_exist_and_take_every_package_import_of_the_suite(self):
+        # The list is kept by hand: the Context reaches runner.py, hw.py
+        # and baseline.py with no import to find. So every package module
+        # a suite module does import has to be on it (or be the registry,
+        # which judges nothing), or that judge would be in no fingerprint.
+        import ast
+        import glob
+        paths = catalog.shared_judge_paths()
+        self.assertEqual([os.path.basename(p) for p in paths], list(catalog.SHARED_JUDGES))
+        for p in paths:
+            self.assertTrue(os.path.isfile(p), p)
+        allowed = set(catalog.SHARED_JUDGES) | {"catalog.py"}
+        for mod in sorted(glob.glob(os.path.join(catalog.suite_dir(), "*.py"))):
+            with open(mod, encoding="utf-8") as f:
+                tree = ast.parse(f.read())
+            for node in tree.body:
+                if isinstance(node, ast.ImportFrom) and node.level == 2:
+                    for name in ([node.module] if node.module else [a.name for a in node.names]):
+                        self.assertIn(name + ".py", allowed,
+                                      "%s imports %s, which no fingerprint carries" % (os.path.basename(mod), name))
+
 
 class ImplementationHashTests(unittest.TestCase):
     """A test's implementation hash is its own function plus the code its
@@ -565,6 +586,32 @@ def two(ctx):
             f.write("def o():\n    return 1\n")
         self.assertEqual((a2, b2), self.shas())
 
+    def test_a_shared_judge_edit_moves_every_test(self):
+        # runner.py, hw.py, baseline.py and puls.py are in every test's
+        # path with no import to follow (the Context, the baseline around
+        # each run), so their text joins every implementation hash.
+        judge = os.path.join(self.tmp, "hwlike.py")
+
+        def write_judge(text):
+            with open(judge, "w", newline="\n") as f:
+                f.write(text)
+            catalog.forget(judge)
+
+        def shas():
+            return tuple(catalog.implementation_sha(self.path, t, judges=[judge]) for t in ("m.one", "m.two"))
+
+        write_judge("def sysfs_int(attr, default=None):\n    return default\n")
+        a1, b1 = shas()
+        write_judge("def sysfs_int(attr, default=None):\n    return 8\n")
+        a2, b2 = shas()
+        self.assertNotEqual(a1, a2)
+        self.assertNotEqual(b1, b2)
+        # a package file outside the list changes nothing
+        other = os.path.join(self.tmp, "server.py")
+        with open(other, "w", newline="\n") as f:
+            f.write("def serve():\n    return 1\n")
+        self.assertEqual((a2, b2), shas())
+
     def test_line_endings_do_not_count(self):
         a1, b1 = self.shas()
         catalog.forget(self.path)