mirror of
https://github.com/openglow-org/forgefirm.git
synced 2026-10-19 02:51:42 -07:00
ScottW514
614efb23e4
workflow_runs counted any simple command in a step's run text as
running the test: _SPLIT_RX split the text on &&, ||, ;, |, newlines,
then, do and else, and every part was matched. So a step that hid the
test's failure, or never ran it, kept a HOST_TESTED row green, against
the HOST_TESTED comment's "a step ... whose failure fails it":
`./build/tray_test || true`, `| tee log`, `&`, `true ||
./build/tray_test`, an `exit 0` or a `set +e` line before the test,
`if false; then ./build/tray_test; fi`, a here-document's lines, a
folded block (`run: >`) whose next line is `|| true`, and a step
under `shell: bash {0}` or a `defaults: run: shell:` of pwsh each
counted. The workflow_runs docstring said "Only what a step's run
commands execute counts", which a command the shell skips, or whose
status it drops, does not meet. No workflow of the tree uses these
forms at a row's test.
workflow_runs now reads the run text line by line as bash -e or sh -e
runs it (_run_commands), a backslash at the end of a line joining the
next. _shell_words reads a line's words and operators, with quotes and
expansions ($(...), ${...}, `...`) inside the words. A command counts
on a line that holds it alone, with its redirections, or in a pipeline
under pipefail (shell: bash, or `set -o pipefail` on an earlier line),
and only while errexit holds: a `set` of errexit or pipefail is
followed. A pipeline with no pipefail does not count. The lint stops at
a line with any other operator (`||`, `&&`, `;`, `&`, a subshell, a
here-document), at a command that opens a compound command or negates a
status (if, case, while, until, a group, a function, `!`), at a command
after which it cannot tell what a later line runs (exit, exec, trap,
eval, source, alias, shopt, hash and the like), and at a `set` of an
option it does not follow (noexec): nothing on that line or after it
counts. A for loop counts in one form: `for <name> in <words>; do` on
its own line, body lines that count (a pipeline under pipefail only),
and `done` on its own line. A step counts only when its own shell:,
and the `defaults: run: shell:` of its job and of the workflow, are
each bash, sh or absent: workflow_steps now reads those defaults, and
reads a folded block, or a scalar continued on the lines under its key,
as one line, as YAML folds it. A prefix is taken off only with the
flags with which it runs its command (`sudo -E`, `env -i`), so `command
-v ./build/tray_test`, which prints the path, does not count; exec is a
stop, no longer a prefix. The HOST_TESTED comment and the docstrings of
workflow_steps and workflow_runs say what the lint reads.
The lint does not read, and counts as no step: `./build/x; true`
(bash -e stops at the test, and the lint refuses the `;` all the
same), `exec ./build/x`, a loop on one line or with `do` on a line of
its own, a pipeline with the test last and no pipefail, a test under
`set +e` whose status a later line checks, a test after a guard line
with `||` or `&&`, and a `set` of any other option (`set -E`). It
counts without reading: what a loop's body does with the loop's name
(its list counts), a `cd` before the test, a `defaults: run:
working-directory:`, what PATH or BASH_ENV change, and the runner's
operating system (it does not read runs-on; a Windows runner's default
shell is pwsh).
Proof: forgetest-ci "Unit tests" (`python -B -m unittest discover -s
tests -v` in forgetest): test_manifest.py runs 39 tests, OK, on Windows
with python 3.14 and on Ubuntu 24.04 with python 3.12.
test_a_step_counts_only_when_every_run_of_the_workflow_runs_it_and_its_failure_fails_the_run
gains these legs, none of which runs the test: `|| true`, `true ||`,
`; true`, `&` and a pipe with no pipefail on the test's line; an `exit
0`, `set +e`, `set -n` or here-document line before it; a `{` group
that `} || true` ends; a loop whose body is `python $t || true`, or a
pipe with no pipefail; a folded block and a continued scalar with `||
true` under the test; a step under `shell: bash {0}`; a job's and a
workflow's `defaults: run: shell: pwsh`. A pipe under `set -o
pipefail` and under shell: bash, and a step under shell: sh, run it.
test_what_running_a_host_test_looks_like gains `command -v
./build/tray_test`, which does not run it; its loop case is the loop as
python3-gfhardware's CI writes it (an assignment from `$(basename "$t"
.py)` in the body), and its glob case is a loop on lines of its own.
Negative controls on scratch copies: manifest.py as it was fails the
first new leg (True is not false), and each new leg reads as running
the test there; 19 mutations of the new reading each fail a case: exit
no stop, errexit not followed, an unknown `set` option followed, a
here-document read as a redirection, no reserved words, pipefail always
on, shell: bash with no pipefail, `set -o pipefail` not followed, the
loop's body not read, a body pipeline with no pipefail read, a folded
block read line by line, a continued scalar's lines dropped, the job's
defaults not read, the workflow's defaults not read, any shell read,
sh not read, `$(` read as an operator, every flag of a prefix taken
off, and `sudo -E` kept. Under bash -e with a failing stand-in test,
the step exits 0 for `|| true`, `true ||`, `&`, a pipe with no
pipefail, an earlier `exit 0`, `set +e` or `set -n`, the `{ } || true`
group, both loop bodies and the folded block; `; true` exits 1. A
here-document's lines are its command's input (`cat <<'EOF'` exits 0),
and `bash {0}` runs with no errexit, so a line after the test decides
the step (exit 0). The 17 workflows of the tree's repositories (211
steps) read to the same events and steps; of their 161 run steps, 20
count fewer commands and 4 the same commands with a redirection taken
off or an expansion kept as written, none of them a step that runs a
row's test. Each
of the 46 HOST_TESTED rows runs its test under the new reading, in the
workflows at the recipe pins and at the heads of forgectrl and
forgeext. The coverage lint on the tree manifest from the recipe pins
reports 0 uncovered paths, 0 host-tested row problems, 43 tests.
pyflakes reports no undefined name.
Acceptance: none. forgetest is dev-only and in no fingerprint (the
source_sha of 43 of 43 tests is unchanged); the lint runs in CI, not in
a campaign, and tests/ are in no fingerprint.
forgetest - the ForgeFIRM release acceptance tool
The daemon behind http://<machine>:8090/ on the dev image: runs the
acceptance catalog against the machine, keeps the append-only result log,
decides which results still apply to the image that is running, exports
the release artifact scripts/release.sh gates on, and serves the bench
diagnostics page. The contract - catalog, campaigns, fingerprints,
inheritance, the gate, the coverage rule - is
the Acceptance page of the documentation site.
Run the host tests
cd forgetest
python3 -m unittest discover -s tests -v
Run the daemon on a workstation (against a mock or a manifest file)
FORGETEST_DATA=/tmp/ft FORGETEST_MANIFEST=../tree-manifest.json \
FORGECTRL_URL=http://<machine> python3 -m forgetest --port 8090
scripts/manifest-from-tree.py produces tree-manifest.json from the recipe
pins; the coverage lint is python3 -m forgetest.coverage --manifest ....
Environment
| Variable | Default | Purpose |
|---|---|---|
FORGETEST_DATA |
/data/forgetest |
results.jsonl, bench.jsonl, token, export/ |
FORGETEST_MANIFEST |
/etc/forgefirm-manifest.json |
the image manifest |
FORGETEST_PORT, FORGETEST_HOST |
8090, 0.0.0.0 | listener |
FORGETEST_BENCH_DIR |
/usr/share/forgetest/bench |
the installed bench scripts |
FORGETEST_BENCH_DATA |
<FORGETEST_DATA>/bench |
passed to bench tools: where they keep their data files (with GF_HOST=127.0.0.1 and the panel token in GF_TOKEN) |
FORGETEST_MARKER |
/run/forgetest.active |
takeover marker |
FORGECTRL_URL, FORGECTRL_TOKEN_FILE |
http://127.0.0.1, /data/forgefirm/panel.token |
forgectrl client (HTTP; the token authorizes writes from the board) |
FORGECTRL_TLS_URL |
https://127.0.0.1 |
forgectrl over HTTPS (self-signed, unverified), for the login test |
GF_SYSFS_ROOT |
/sys/glowforge/ |
kernel module sysfs |
GRBL_HOST, GRBL_PORT |
127.0.0.1, 23 | Grbl TCP |
Adding a test
Register it in the subsystem module under forgetest/suite/ with
@test(...): id subsystem.name, kind, hardware, mode (the controller
mode the test needs; the runner switches to it first), covers,
requires, always, steps. The body gets a Context (log, check, fail,
prompt, confirm, instruct, sleep, evidence, forgectrl, sysfs,
grbl, takeover). Return normally for PASS, raise runner.Failed for
FAIL. Then run the unit tests and the coverage lint.