mirror of
https://github.com/openglow-org/forgefirm.git
synced 2026-10-19 02:51:42 -07:00
forgetest: the shared judges join every test's implementation hash
Every catalog test judges the machine through package modules it never imports: the Context a test is handed and the takeover (runner.py), the sysfs, LED, switch and HTTP readers behind it (hw.py), and the fresh-boot idle state judged around every run (baseline.py). The cloud and extension tests print the dark job of puls.py, which the suite imports at level 2, and the sibling walk follows level-1 imports only. So none of the four was in any fingerprint: a weakening of sysfs_int, button_lit, Context.check, wait_for or position_quantized moved no hash, and an inherited PASS of kernel.latch-locked-idle, laser.armed-kill or any takeover test survived it unseen by the gate. catalog.py names them in SHARED_JUDGES, with the criterion in the comment above it: a module every test runs through without importing it, or the dark job the cloud tests print. The rest of the package (the server, the page, the campaign rules, the artifact, the manifest, the registry, the bench actuator) judges nothing about the machine and stays out. shared_judge_paths() resolves the four next to the module, and implementation_sha folds their text after the module's shared text and its imported siblings. A listed file that is missing raises, never a hash computed without it. _direct_siblings stays level-1 only, and the tool version does not move: the gate judges by source_sha and the fingerprint. Every catalog test's implementation hash moves once with this change, so every recorded PASS is stale and the next campaign is a full one, 43 tests. The catalog hash does not move, so an open campaign stays open with every test required. From here on, an edit to runner.py, hw.py, baseline.py or puls.py moves every test. Proof: forgetest-ci "Unit tests" (`python -B -m unittest discover -s tests -v` in forgetest): test_manifest.py runs 38 tests, OK, with test_a_shared_judge_edit_moves_every_test (an edit to a judge moves both tests of a module, a package file outside the list moves neither) and test_the_shared_judges_exist_and_take_every_package_import_of_the_suite (the four files exist next to catalog.py, and every level-2 import of a suite module is a listed judge or the registry). Negative controls on scratch copies: `judged = []` in implementation_sha fails the first case alone; "runner.py" dropped from SHARED_JUDGES fails the guard alone, naming homeoff.py's import of runner. On the real catalog the source_sha of 43 of 43 tests differs from HEAD's; with hw.sysfs_int weakened to return the latch bit on an unparsable read, 0 of 43 move at HEAD and 43 of 43 move with this change. The coverage lint on the tree manifest from the recipe pins reports 0 uncovered paths, 0 host-tested row problems, 43 tests. pyflakes reports no undefined name. Acceptance: every catalog test. The implementation hash of all 43 moves once, so the next campaign runs the whole catalog.
This commit is contained in:
@@ -479,6 +479,27 @@ class CatalogTests(unittest.TestCase):
|
||||
for t in catalog.all_tests(reg):
|
||||
self.assertEqual(len(t.source_sha), 64)
|
||||
|
||||
def test_the_shared_judges_exist_and_take_every_package_import_of_the_suite(self):
|
||||
# The list is kept by hand: the Context reaches runner.py, hw.py
|
||||
# and baseline.py with no import to find. So every package module
|
||||
# a suite module does import has to be on it (or be the registry,
|
||||
# which judges nothing), or that judge would be in no fingerprint.
|
||||
import ast
|
||||
import glob
|
||||
paths = catalog.shared_judge_paths()
|
||||
self.assertEqual([os.path.basename(p) for p in paths], list(catalog.SHARED_JUDGES))
|
||||
for p in paths:
|
||||
self.assertTrue(os.path.isfile(p), p)
|
||||
allowed = set(catalog.SHARED_JUDGES) | {"catalog.py"}
|
||||
for mod in sorted(glob.glob(os.path.join(catalog.suite_dir(), "*.py"))):
|
||||
with open(mod, encoding="utf-8") as f:
|
||||
tree = ast.parse(f.read())
|
||||
for node in tree.body:
|
||||
if isinstance(node, ast.ImportFrom) and node.level == 2:
|
||||
for name in ([node.module] if node.module else [a.name for a in node.names]):
|
||||
self.assertIn(name + ".py", allowed,
|
||||
"%s imports %s, which no fingerprint carries" % (os.path.basename(mod), name))
|
||||
|
||||
|
||||
class ImplementationHashTests(unittest.TestCase):
|
||||
"""A test's implementation hash is its own function plus the code its
|
||||
@@ -565,6 +586,32 @@ def two(ctx):
|
||||
f.write("def o():\n return 1\n")
|
||||
self.assertEqual((a2, b2), self.shas())
|
||||
|
||||
def test_a_shared_judge_edit_moves_every_test(self):
|
||||
# runner.py, hw.py, baseline.py and puls.py are in every test's
|
||||
# path with no import to follow (the Context, the baseline around
|
||||
# each run), so their text joins every implementation hash.
|
||||
judge = os.path.join(self.tmp, "hwlike.py")
|
||||
|
||||
def write_judge(text):
|
||||
with open(judge, "w", newline="\n") as f:
|
||||
f.write(text)
|
||||
catalog.forget(judge)
|
||||
|
||||
def shas():
|
||||
return tuple(catalog.implementation_sha(self.path, t, judges=[judge]) for t in ("m.one", "m.two"))
|
||||
|
||||
write_judge("def sysfs_int(attr, default=None):\n return default\n")
|
||||
a1, b1 = shas()
|
||||
write_judge("def sysfs_int(attr, default=None):\n return 8\n")
|
||||
a2, b2 = shas()
|
||||
self.assertNotEqual(a1, a2)
|
||||
self.assertNotEqual(b1, b2)
|
||||
# a package file outside the list changes nothing
|
||||
other = os.path.join(self.tmp, "server.py")
|
||||
with open(other, "w", newline="\n") as f:
|
||||
f.write("def serve():\n return 1\n")
|
||||
self.assertEqual((a2, b2), shas())
|
||||
|
||||
def test_line_endings_do_not_count(self):
|
||||
a1, b1 = self.shas()
|
||||
catalog.forget(self.path)
|
||||
|
||||
Reference in New Issue
Block a user