ScottW514
4d8f0d2e67
A HOST_TESTED row names a workflow and a host test, and the coverage lint judges the row by what the workflow's steps run. The parser stored a step's `if:` and `continue-on-error:` and read neither, read no job key, and read no `on:`. So a step under `if: false`, a step whose failure the run tolerates, a job under an `if:`, and a workflow on `workflow_dispatch` alone all counted as running the test, and a checkout under a condition counted as a checkout: a disabled or tolerated host test keeps its row green and its paths covered. The lint now reads the workflow's events and the conditions of a step and of its job. `workflow_triggers` returns the events of the `on:` mapping, list or scalar. What is under an event (its branches, its paths, a dispatch's inputs) is not an event, and a flow mapping names none. A row's workflow must run on push or pull_request, or the lint reports that it runs on no push or pull request: a dispatch or a schedule alone runs on no change, and a call alone is listed as well, since the lint does not follow it to its caller. `workflow_steps` tracks the job each step is in and carries the job's `if:` and `continue-on-error:` on the step as `job`. `_on_every_run` counts a step only when neither it nor its job carries an `if:` other than the literal `true` or a `continue-on-error:` other than the literal `false`. `workflow_runs` and `checkout_path` use it, so a conditional step runs nothing and a conditional checkout is no checkout. The reason for such a step stays "does not run it". `needs:` is not read, nor are the `paths:` and `branches:` filters of an event: the lint stays a floor there. Proof: forgetest-ci "Unit tests" (`python -B -m unittest discover -s tests -v` in forgetest): test_manifest.py runs 36 tests, OK, with the two new cases, test_a_step_counts_only_when_every_run_of_the_workflow_runs_it_and_its_failure_fails_the_run and test_the_events_a_workflow_runs_on, and with test_the_workflow_has_to_run_the_host_test extended by a dispatch-only workflow and a tolerated step. Negative controls on scratch copies: `_on_every_run` forced to return True fails the step case and the host-test case (failures=2); the trigger check deleted from host_row_problems fails the host-test case (failures=1). The coverage lint on the tree manifest from the recipe pins reports 0 uncovered paths, 0 host-tested row problems, 43 tests. pyflakes reports no undefined name. Acceptance: none. forgetest is dev-only and in no fingerprint; the lint runs in CI, not in a campaign.
forgetest - the ForgeFIRM release acceptance tool
The daemon behind http://<machine>:8090/ on the dev image: runs the
acceptance catalog against the machine, keeps the append-only result log,
decides which results still apply to the image that is running, exports
the release artifact scripts/release.sh gates on, and serves the bench
diagnostics page. The contract - catalog, campaigns, fingerprints,
inheritance, the gate, the coverage rule - is
the Acceptance page of the documentation site.
Run the host tests
cd forgetest
python3 -m unittest discover -s tests -v
Run the daemon on a workstation (against a mock or a manifest file)
FORGETEST_DATA=/tmp/ft FORGETEST_MANIFEST=../tree-manifest.json \
FORGECTRL_URL=http://<machine> python3 -m forgetest --port 8090
scripts/manifest-from-tree.py produces tree-manifest.json from the recipe
pins; the coverage lint is python3 -m forgetest.coverage --manifest ....
Environment
| Variable | Default | Purpose |
|---|---|---|
FORGETEST_DATA |
/data/forgetest |
results.jsonl, bench.jsonl, token, export/ |
FORGETEST_MANIFEST |
/etc/forgefirm-manifest.json |
the image manifest |
FORGETEST_PORT, FORGETEST_HOST |
8090, 0.0.0.0 | listener |
FORGETEST_BENCH_DIR |
/usr/share/forgetest/bench |
the installed bench scripts |
FORGETEST_BENCH_DATA |
<FORGETEST_DATA>/bench |
passed to bench tools: where they keep their data files (with GF_HOST=127.0.0.1 and the panel token in GF_TOKEN) |
FORGETEST_MARKER |
/run/forgetest.active |
takeover marker |
FORGECTRL_URL, FORGECTRL_TOKEN_FILE |
http://127.0.0.1, /data/forgefirm/panel.token |
forgectrl client (HTTP; the token authorizes writes from the board) |
FORGECTRL_TLS_URL |
https://127.0.0.1 |
forgectrl over HTTPS (self-signed, unverified), for the login test |
GF_SYSFS_ROOT |
/sys/glowforge/ |
kernel module sysfs |
GRBL_HOST, GRBL_PORT |
127.0.0.1, 23 | Grbl TCP |
Adding a test
Register it in the subsystem module under forgetest/suite/ with
@test(...): id subsystem.name, kind, hardware, mode (the controller
mode the test needs; the runner switches to it first), covers,
requires, always, steps. The body gets a Context (log, check, fail,
prompt, confirm, instruct, sleep, evidence, forgectrl, sysfs,
grbl, takeover). Return normally for PASS, raise runner.Failed for
FAIL. Then run the unit tests and the coverage lint.