mirror of
https://github.com/openglow-org/forgefirm.git
synced 2026-10-17 10:01:41 -07:00
ScottW514
b90299acdc
Four catalog tests leave for the host proofs that carry them; the
catalog goes from 87 to 83 (83 @test in forgetest/forgetest/suite/).
Each proof passed; none skipped.
- exthost.package-routes, for forgectrl's tests/host_ext.sh (05848ac):
against a compiled stand-in that is a live host, GET /ext/status and
POST /ext/package are 403 without the token with the host not run,
the status passes the host's list through with host.running true and
the stand-in's pid, enable runs as the host's command, a bad action
and a bad id are 400 with nothing run, and a package that is not
installed is 409 in the host's words. extpkg_test holds the relay's
argv; the state file, the required-holds directory and removal are
forgeext's install_test.py and run_test.py; the page card is
check_ui_js.py. REQUIRED_HOLDS, which only it read, and the "the hold"
section header over it go.
- exthost.wizard, for host_ext.sh (05848ac): the check listed, run to its
prompt with no lease holder, answered yes, ended with its summary and
listed done; the setup record's wizards and the gate as they were; a
second run aborted; a check of a package that is not installed 404.
wizpkg_test holds each step to its form. suite/extwizard.py held only
it: the module and its import go.
- exthost.catalog: install_test.py and extpkg_test already assert its
behavioral claims, and the one fact the bench test did not prove, that
the image's extension key is the catalog's published key, is now a
forgetest-ci step: the catalog is checked out at main as a sibling and
meta-forgefirm/recipes-forgefirm/forgefirm-keys/files/ext/forgefirm-ext.pub
is cmp -s'd against its keys/forgefirm-ext.pub, either one missing a
failure. suite/extcat.py held only it: the module and its import go.
- camera.key-read, for forgectrl's tests/host_lan.sh (05848ac): with
panel_open_reads=0, GET /cam/status from the LAN client over HTTP and
HTTPS is 403 with no key or a wrong one and 200 with the key as ?key=
or the header, the key alone writes no setting, and a rotation stops
the old key and the new one reads. camera.py's json import, which only
it used, goes.
The coverage lint then named forgectrl src/wizpkg.c, src/wizpkg.h,
src/camkey.c and src/camkey.h, and nothing else. Two HOST_TESTED rows
take them, to forgectrl:tests/wizpkg_test.c and
forgectrl:tests/camkey_test.c on forgectrl:.github/workflows/build.yml,
which the pin c274f19 runs (./build/wizpkg_test, ./build/camkey_test).
src/extpkg.* stays covered (logs.tree-tail-export, logs.routing), and
forgeext/** by exthost.service.
The key step's run text, taken from the workflow, passes from the tree
root with both files present; controls on a /tmp copy, a published key
that differs and one that is missing: each fails. test_manifest.py (34)
passes. The coverage lint on the tree manifest finds no uncovered path
and no host-tested row problem across the 83 tests; its one finding is
setup.check-lens's empty src/lensfind.* entry. pyflakes reports nothing
on exthost.py, camera.py or manifest.py. The full forgetest suite was
not run.
Fingerprints, measured: 10 tests move, the 5 left in exthost.py
(armed-freeze, motion-jog, motion-job, platform, service) and the 5 left
in camera.py (frame-health, h264-stream, lid-privacy, sensor-profile,
snapshot). The pending pin bumps from forgectrl c274f19 and forgeext
2d0391e already owe 9 of them; exthost.platform is owed already since
30c0a05. This adds none. releases/ and TEST-AUDIT.md are unchanged.
112 lines
4.4 KiB
YAML
112 lines
4.4 KiB
YAML
# forgetest: the release acceptance tool's host-side proof.
|
|
#
|
|
# - unit tests: campaign rules, fingerprints, artifact build/verify (the
|
|
# release gate's decision, including the negative fixtures), runner +
|
|
# HTTP API end to end with a fake catalog and a fake bench tool, the
|
|
# suites replayed on the machine's own log lines, and the check that
|
|
# every log phrase the cloud suite greps for is one the pinned cloud
|
|
# app can log (it reads the app sources from the manifest cache)
|
|
# - the source bundle of a release: which recipe of the image needs
|
|
# source, which archive answers for it, and the refusal that stops a
|
|
# release with source missing
|
|
# - shared UI files: theme.css and the vendored Bootstrap the page
|
|
# carries are byte-identical to forgectrl's at its pinned revision
|
|
# - the extension key: the public key the image installs for the official
|
|
# tier is byte-identical to the one the extension catalog publishes
|
|
# (keys/forgefirm-ext.pub on its main branch)
|
|
# - coverage lint: every source path of every component pinned by the
|
|
# recipes must be selected by some catalog test's coverage globs or by a
|
|
# host-tested row naming the host test that proves it (the tree manifest
|
|
# is generated from the pins with git - no Yocto build); enforced: an
|
|
# uncovered path fails the job (https://docs.forgefirm.org/developers/testing/).
|
|
|
|
name: forgetest-ci
|
|
|
|
on:
|
|
push:
|
|
branches: [master]
|
|
paths:
|
|
- 'forgetest/**'
|
|
- 'scripts/acceptance-gate.py'
|
|
- 'scripts/manifest-from-tree.py'
|
|
- 'scripts/source-bundle.py'
|
|
- 'kas/source-bundle.yml'
|
|
- 'meta-forgefirm/recipes-forgefirm/**'
|
|
- '.github/workflows/forgetest-ci.yml'
|
|
pull_request:
|
|
workflow_dispatch:
|
|
|
|
jobs:
|
|
test:
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 20
|
|
steps:
|
|
- name: Checkout forgefirm
|
|
uses: actions/checkout@v4
|
|
with:
|
|
path: forgefirm
|
|
|
|
- name: Checkout meta-openglow (sibling)
|
|
uses: actions/checkout@v4
|
|
with:
|
|
repository: openglow-org/meta-openglow
|
|
ref: scarthgap
|
|
path: meta-openglow
|
|
|
|
- name: Checkout forgefirm-extensions-catalog (sibling)
|
|
uses: actions/checkout@v4
|
|
with:
|
|
repository: openglow-org/forgefirm-extensions-catalog
|
|
ref: main
|
|
path: forgefirm-extensions-catalog
|
|
|
|
- name: The image's extension key is the catalog's published key
|
|
run: |
|
|
image=forgefirm/meta-forgefirm/recipes-forgefirm/forgefirm-keys/files/ext/forgefirm-ext.pub
|
|
published=forgefirm-extensions-catalog/keys/forgefirm-ext.pub
|
|
for f in "$image" "$published"; do
|
|
test -f "$f" || { echo "missing: $f"; exit 1; }
|
|
done
|
|
cmp -s "$image" "$published" || { echo "$image is not $published"; exit 1; }
|
|
|
|
- name: Set up Python
|
|
uses: actions/setup-python@v5
|
|
with:
|
|
python-version: '3.12'
|
|
|
|
- name: Tree manifest from the recipe pins (also fetches the pinned app sources the needle check reads)
|
|
working-directory: forgefirm
|
|
run: python scripts/manifest-from-tree.py --out tree-manifest.json
|
|
|
|
- name: Unit tests
|
|
working-directory: forgefirm/forgetest
|
|
# -B: no bytecode caches under the package directory (the recipe's fetch
|
|
# checksum would otherwise move without a source change)
|
|
run: python -B -m unittest discover -s tests -v
|
|
|
|
- name: No undefined names (the live drills never execute on the host)
|
|
working-directory: forgefirm/forgetest
|
|
run: |
|
|
pip install pyflakes
|
|
if python -B -m pyflakes forgetest tests | grep -E "undefined name"; then exit 1; fi
|
|
|
|
- name: Shared UI files match forgectrl at its pinned revision
|
|
working-directory: forgefirm
|
|
run: python scripts/check-ui-vendor.py
|
|
|
|
- name: Coverage lint (enforced)
|
|
working-directory: forgefirm/forgetest
|
|
run: python -B -m forgetest.coverage --manifest ../tree-manifest.json --enforce | tee "$GITHUB_STEP_SUMMARY"
|
|
|
|
- name: Gate self-check on the tree manifest
|
|
working-directory: forgefirm
|
|
run: |
|
|
# The gate must load the catalog and refuse a missing/unauthorized
|
|
# artifact cleanly (exit 1/2, never a traceback).
|
|
set +e
|
|
echo '{}' > empty.json
|
|
python scripts/acceptance-gate.py empty.json tree-manifest.json --quiet
|
|
rc=$?
|
|
set -e
|
|
test "$rc" -eq 1
|