ScottW514
c717ecbcce
_KEY_RX took an opening quote and a closing quote each on its own, so a key with one stray quote read as the bare key: `push":` and `pull_request':` counted as push and pull_request, where YAML reads the keys `push"` and `pull_request'`, which are no events; a step's `run": ...` counted as its run command, and `uses": actions/checkout@v4` as a checkout. A HOST_TESTED row whose workflow was spelled so passed the lint, while that workflow runs on no push and runs no such step. _KEY_RX now reads a quote only as a matched pair. A key with one stray quote is no key the lint reads, and where the lint reads keys it counts as running nothing: an event key spelled so makes the `on:` name no event, and one at the top level, at a job's name or key, or at a step's key below its dash line makes every step of the workflow count as no step (each carries "unread", which _on_every_run refuses). So a step's `if": false` still keeps the step from counting, as it did under the loose reading, and under a `jobs":` top level, where a job's `if: false` goes unread, no step counts either. The HOST_TESTED comment says so. The lint only lists more: a workflow with no such key reads as before. Proof: forgetest-ci "Unit tests" (`python -B -m unittest discover -s tests -v` in forgetest): test_manifest.py runs 39 tests, OK, on Windows with python 3.14 and on Ubuntu 24.04 with python 3.12. test_the_events_a_workflow_runs_on gains an `on:` with `push":`, and one with `push:` and `'pull_request:`, each naming no event, and test_a_step_counts_only_when_every_run_of_the_workflow_runs_it_and_its_failure_fails_the_run gains a step with `if": false`, a step with `run": ./build/tray_test`, a job with `if": false` and a `jobs":` top level, none of them running the test. Negative controls on scratch copies: _KEY_RX as it was fails the `run":` step and the `push":` event; the matched pairs with no stray-quote rule fail the `if": false` step and the second `on:`; the rule at a step's keys alone fails the job's `if": false`. The 17 workflows of the tree's repositories (211 steps) read to the same events and steps before and after the change. The coverage lint on the tree manifest from the recipe pins reports 0 uncovered paths, 0 host-tested row problems, 43 tests. pyflakes reports no undefined name. Acceptance: none. forgetest is dev-only and in no fingerprint (the source_sha of 43 of 43 tests is unchanged); the lint runs in CI, not in a campaign, and tests/ are in no fingerprint.
forgetest - the ForgeFIRM release acceptance tool
The daemon behind http://<machine>:8090/ on the dev image: runs the
acceptance catalog against the machine, keeps the append-only result log,
decides which results still apply to the image that is running, exports
the release artifact scripts/release.sh gates on, and serves the bench
diagnostics page. The contract - catalog, campaigns, fingerprints,
inheritance, the gate, the coverage rule - is
the Acceptance page of the documentation site.
Run the host tests
cd forgetest
python3 -m unittest discover -s tests -v
Run the daemon on a workstation (against a mock or a manifest file)
FORGETEST_DATA=/tmp/ft FORGETEST_MANIFEST=../tree-manifest.json \
FORGECTRL_URL=http://<machine> python3 -m forgetest --port 8090
scripts/manifest-from-tree.py produces tree-manifest.json from the recipe
pins; the coverage lint is python3 -m forgetest.coverage --manifest ....
Environment
| Variable | Default | Purpose |
|---|---|---|
FORGETEST_DATA |
/data/forgetest |
results.jsonl, bench.jsonl, token, export/ |
FORGETEST_MANIFEST |
/etc/forgefirm-manifest.json |
the image manifest |
FORGETEST_PORT, FORGETEST_HOST |
8090, 0.0.0.0 | listener |
FORGETEST_BENCH_DIR |
/usr/share/forgetest/bench |
the installed bench scripts |
FORGETEST_BENCH_DATA |
<FORGETEST_DATA>/bench |
passed to bench tools: where they keep their data files (with GF_HOST=127.0.0.1 and the panel token in GF_TOKEN) |
FORGETEST_MARKER |
/run/forgetest.active |
takeover marker |
FORGECTRL_URL, FORGECTRL_TOKEN_FILE |
http://127.0.0.1, /data/forgefirm/panel.token |
forgectrl client (HTTP; the token authorizes writes from the board) |
FORGECTRL_TLS_URL |
https://127.0.0.1 |
forgectrl over HTTPS (self-signed, unverified), for the login test |
GF_SYSFS_ROOT |
/sys/glowforge/ |
kernel module sysfs |
GRBL_HOST, GRBL_PORT |
127.0.0.1, 23 | Grbl TCP |
Adding a test
Register it in the subsystem module under forgetest/suite/ with
@test(...): id subsystem.name, kind, hardware, mode (the controller
mode the test needs; the runner switches to it first), covers,
requires, always, steps. The body gets a Context (log, check, fail,
prompt, confirm, instruct, sleep, evidence, forgectrl, sysfs,
grbl, takeover). Return normally for PASS, raise runner.Failed for
FAIL. Then run the unit tests and the coverage lint.