mirror of
https://github.com/openglow-org/forgefirm.git
synced 2026-09-28 09:11:11 -07:00
ScottW514
d482e76402
A field install failed on "firmware download failed", and worked after a reboot. The download was one bare curl -fL: no retry, no resume, no bound on a stalled transfer, and nothing on the machine recorded what had gone wrong. The download. download_fw makes up to five tries, 5, 15, 30 and 60 seconds apart. Each try resumes the partial file (curl -C -) and is bounded: 20 s to connect, and a transfer below 1 KB/s for 30 s ends the try. The file is written as forgefirm.fw.part and takes its name only when curl finished; the signature check that follows is what vouches for its content. A full disk (curl 23) and a release that is not there (HTTP 404) end the tries at once, because waiting cannot fix them. A partial file the server will not resume (curl 33 or 36, HTTP 416) starts over. The loop is the installer's own rather than curl --retry: the factory curl on the bench reference is 7.69.1, whose --retry does not count a resolver failure or a dropped transfer as retryable, and older factory builds carry older curls. The owner sees the reason in words with each retry, and the final failure says that a re-run goes straight to the download, because the archives are kept. The log. Every run appends to /data/log/forgefirm/install/install.log, in the log tree's own line format (UTC, program "install"): the installer's md5 (which revision ran), the factory version and the slots, the owner's answers, each archive, each download try with curl's exit code, the HTTP code and the reason, the machine's clock at each try (a wrong clock breaks TLS), and after a failed try the address, the default route, the resolver and whether github.com resolves; then the signature and identity checks, the write, the boot selection, and the reason for any failure through die(). The log is appended across runs, so the run that failed is still there after the run that worked. Logging never fails the install. forgectrl's log export carries the directory (forgectrl 0dae758). Proven: tests/test_installer.py runs the installer's own functions under sh against a scripted curl - a clean download, a resolver failure and a dropped transfer that resume to the full file, the tries running out, 404 and a full disk ending them at once, a stale partial file starting over, the TLS reason naming the clock, every log line in the tree format, die() leaving its reason, and an unwritable log not failing the run. The whole host suite, 409 tests, passes under Linux and the coverage lint is clean. Bench: the same functions under the factory firmware's own shell (busybox 1.31.1 ash, the factory slot of the bench reference in a chroot) resumed, retried, ran out of tries and logged exactly as under sh. Acceptance: logs.tree-tail-export now plants a probe file in the install directory and requires it back in the export bundle, its line intact and its MAC and IPv4 address redacted, and requires an install log in the bundle when the machine has one. The installer itself is not on the image: the install page fetches it from master, so it is live with this push.
forgetest - the ForgeFIRM release acceptance tool
The daemon behind http://<machine>:8090/ on the dev image: runs the
acceptance catalog against the machine, keeps the append-only result log,
decides which results still apply to the image that is running, exports
the release artifact scripts/release.sh gates on, and serves the bench
diagnostics page. The contract - catalog, campaigns, fingerprints,
inheritance, the gate, the coverage rule - is
the Acceptance page of the documentation site.
Run the host tests
cd forgetest
python3 -m unittest discover -s tests -v
Run the daemon on a workstation (against a mock or a manifest file)
FORGETEST_DATA=/tmp/ft FORGETEST_MANIFEST=../tree-manifest.json \
FORGECTRL_URL=http://<machine> python3 -m forgetest --port 8090
scripts/manifest-from-tree.py produces tree-manifest.json from the recipe
pins; the coverage lint is python3 -m forgetest.coverage --manifest ....
Environment
| Variable | Default | Purpose |
|---|---|---|
FORGETEST_DATA |
/data/forgetest |
results.jsonl, bench.jsonl, token, export/ |
FORGETEST_MANIFEST |
/etc/forgefirm-manifest.json |
the image manifest |
FORGETEST_PORT, FORGETEST_HOST |
8090, 0.0.0.0 | listener |
FORGETEST_BENCH_DIR |
/usr/share/forgetest/bench |
the installed bench scripts |
FORGETEST_BENCH_DATA |
<FORGETEST_DATA>/bench |
passed to bench tools: where they keep their data files (with GF_HOST=127.0.0.1 and the panel token in GF_TOKEN) |
FORGETEST_MARKER |
/run/forgetest.active |
takeover marker |
FORGECTRL_URL, FORGECTRL_TOKEN_FILE |
http://127.0.0.1, /data/forgefirm/panel.token |
forgectrl client (HTTP; the token authorizes writes from the board) |
FORGECTRL_TLS_URL |
https://127.0.0.1 |
forgectrl over HTTPS (self-signed, unverified), for the login test |
GF_SYSFS_ROOT |
/sys/glowforge/ |
kernel module sysfs |
GRBL_HOST, GRBL_PORT |
127.0.0.1, 23 | Grbl TCP |
Adding a test
Register it in the subsystem module under forgetest/suite/ with
@test(...): id subsystem.name, kind, hardware, mode (the controller
mode the test needs; the runner switches to it first), covers,
requires, always, steps. The body gets a Context (log, check, fail,
prompt, confirm, instruct, sleep, evidence, forgectrl, sysfs,
grbl, takeover). Return normally for PASS, raise runner.Failed for
FAIL. Then run the unit tests and the coverage lint.