mirror of
https://github.com/openglow-org/forgefirm.git
synced 2026-10-19 02:51:42 -07:00
ScottW514
850498fb00
Every catalog test judges the machine through package modules it never imports: the Context a test is handed and the takeover (runner.py), the sysfs, LED, switch and HTTP readers behind it (hw.py), and the fresh-boot idle state judged around every run (baseline.py). The cloud and extension tests print the dark job of puls.py, which the suite imports at level 2, and the sibling walk follows level-1 imports only. So none of the four was in any fingerprint: a weakening of sysfs_int, button_lit, Context.check, wait_for or position_quantized moved no hash, and an inherited PASS of kernel.latch-locked-idle, laser.armed-kill or any takeover test survived it unseen by the gate. catalog.py names them in SHARED_JUDGES, with the criterion in the comment above it: a module every test runs through without importing it, or the dark job the cloud tests print. The rest of the package (the server, the page, the campaign rules, the artifact, the manifest, the registry, the bench actuator) judges nothing about the machine and stays out. shared_judge_paths() resolves the four next to the module, and implementation_sha folds their text after the module's shared text and its imported siblings. A listed file that is missing raises, never a hash computed without it. _direct_siblings stays level-1 only, and the tool version does not move: the gate judges by source_sha and the fingerprint. Every catalog test's implementation hash moves once with this change, so every recorded PASS is stale and the next campaign is a full one, 43 tests. The catalog hash does not move, so an open campaign stays open with every test required. From here on, an edit to runner.py, hw.py, baseline.py or puls.py moves every test. Proof: forgetest-ci "Unit tests" (`python -B -m unittest discover -s tests -v` in forgetest): test_manifest.py runs 38 tests, OK, with test_a_shared_judge_edit_moves_every_test (an edit to a judge moves both tests of a module, a package file outside the list moves neither) and test_the_shared_judges_exist_and_take_every_package_import_of_the_suite (the four files exist next to catalog.py, and every level-2 import of a suite module is a listed judge or the registry). Negative controls on scratch copies: `judged = []` in implementation_sha fails the first case alone; "runner.py" dropped from SHARED_JUDGES fails the guard alone, naming homeoff.py's import of runner. On the real catalog the source_sha of 43 of 43 tests differs from HEAD's; with hw.sysfs_int weakened to return the latch bit on an unparsable read, 0 of 43 move at HEAD and 43 of 43 move with this change. The coverage lint on the tree manifest from the recipe pins reports 0 uncovered paths, 0 host-tested row problems, 43 tests. pyflakes reports no undefined name. Acceptance: every catalog test. The implementation hash of all 43 moves once, so the next campaign runs the whole catalog.
forgetest - the ForgeFIRM release acceptance tool
The daemon behind http://<machine>:8090/ on the dev image: runs the
acceptance catalog against the machine, keeps the append-only result log,
decides which results still apply to the image that is running, exports
the release artifact scripts/release.sh gates on, and serves the bench
diagnostics page. The contract - catalog, campaigns, fingerprints,
inheritance, the gate, the coverage rule - is
the Acceptance page of the documentation site.
Run the host tests
cd forgetest
python3 -m unittest discover -s tests -v
Run the daemon on a workstation (against a mock or a manifest file)
FORGETEST_DATA=/tmp/ft FORGETEST_MANIFEST=../tree-manifest.json \
FORGECTRL_URL=http://<machine> python3 -m forgetest --port 8090
scripts/manifest-from-tree.py produces tree-manifest.json from the recipe
pins; the coverage lint is python3 -m forgetest.coverage --manifest ....
Environment
| Variable | Default | Purpose |
|---|---|---|
FORGETEST_DATA |
/data/forgetest |
results.jsonl, bench.jsonl, token, export/ |
FORGETEST_MANIFEST |
/etc/forgefirm-manifest.json |
the image manifest |
FORGETEST_PORT, FORGETEST_HOST |
8090, 0.0.0.0 | listener |
FORGETEST_BENCH_DIR |
/usr/share/forgetest/bench |
the installed bench scripts |
FORGETEST_BENCH_DATA |
<FORGETEST_DATA>/bench |
passed to bench tools: where they keep their data files (with GF_HOST=127.0.0.1 and the panel token in GF_TOKEN) |
FORGETEST_MARKER |
/run/forgetest.active |
takeover marker |
FORGECTRL_URL, FORGECTRL_TOKEN_FILE |
http://127.0.0.1, /data/forgefirm/panel.token |
forgectrl client (HTTP; the token authorizes writes from the board) |
FORGECTRL_TLS_URL |
https://127.0.0.1 |
forgectrl over HTTPS (self-signed, unverified), for the login test |
GF_SYSFS_ROOT |
/sys/glowforge/ |
kernel module sysfs |
GRBL_HOST, GRBL_PORT |
127.0.0.1, 23 | Grbl TCP |
Adding a test
Register it in the subsystem module under forgetest/suite/ with
@test(...): id subsystem.name, kind, hardware, mode (the controller
mode the test needs; the runner switches to it first), covers,
requires, always, steps. The body gets a Context (log, check, fail,
prompt, confirm, instruct, sleep, evidence, forgectrl, sysfs,
grbl, takeover). Return normally for PASS, raise runner.Failed for
FAIL. Then run the unit tests and the coverage lint.