diff --git a/docs/BRINGUP.md b/docs/BRINGUP.md index 2662087..05eefa8 100644 --- a/docs/BRINGUP.md +++ b/docs/BRINGUP.md @@ -1371,8 +1371,13 @@ feature requests, enhancements) will eventually be tracked as GitHub issues. factory recovery after the 60 s watchdog with a power cycle returning, as documented. Image 20260902230436 (release and dev) is built on the pushed pins, the kernel and the module rebuilt together for the watchdog sysfs - change. Owed now: flash the dev image, take a fresh-boot baseline, and run - the full campaign on it, the campaign the release gate asks for. + change, and is not the image the campaign runs on: the campaign waits + until every audit finding, the deferred ones included, is on one image + (the operator's rule). That batch is done (item 11) and rides the next + local image, which is the campaign's. Owed now: build it, flash the dev + image, take a fresh-boot baseline, run the full campaign on it, the + campaign the release gate asks for; then the push in CI order and the + pin bumps for the batch. Decisions taken in the remediation that the operator confirms or reverses: the release image has no shell login (the install page now says so); the cloud client holds and resumes on the cooling verdict with a 30-minute @@ -1390,36 +1395,33 @@ feature requests, enhancements) will eventually be tracked as GitHub issues. change: it rides the next image flash, with the coolant-reading tests (`cooling.aa-offset-calibrate`, `cooling.flow-verify`) as its proof. -11. **The audit's deferred findings.** Six findings of the 2026-09-01 audit were - deferred with a reason and are owed here. Four are one kernel batch, since - each edits the SDMA script, the module, or a kernel patch file, and a - kernel change rides one image flash: **K-4** (end-of-data reached before - an armed waypoint is decoded as the waypoint; the script clears its - scratch at `alldone` and publishes an end-of-data flag the host decodes on; - proof: a cloud pause within the last bytes of a job, a `cloud.pause-resume` - case), **K-8** (the FIRE restore at the resume waypoint writes the whole - GPIO2 data register from gpiolib's shadow while the script may be - mid-byte; move the restore into the script on a scratch flag; a scope on - DIR at the resume point first, since the failure is plausible, not seen), - **P-13** (the SDMA glue patch hands out a channel without claiming it from - dmaengine and re-initializes a tasklet that may be scheduled; claim the - channel in `sdma_get_channel()`, release it in the put, `tasklet_kill()` - before clearing; latent today, no other client reaches the channel), and - **P-14** (the SPI patch takes its inter-word wait states from - `spi_transfer.delay`, which the core also applies after the transfer; - read `word_delay` and have `pic.c` set it; batch with item 10, since the - PIC read pattern measured on 2026-09-02 may turn on these wait states). - The patch-file edits are regenerated with `devtool modify linux-fslc` on - the pinned tree. Two are host-only: **B-16** (a host `__pycache__` under the - forgetest package directory moves the recipe's task hash without a source - change; the remediation named the package directory and drops the caches - in `do_unpack`, but the fetch checksum is taken before that, so the - residual stays: run the host tests with `PYTHONDONTWRITEBYTECODE=1` or - clear the caches before a build, since the file fetcher has no exclude); - and **FA-20** (the panel's devserver mock diverged from the daemon: - mode values, `/cool/status`, the gate table, the settings keys, the diag - routes; regenerate its tables from `gates.c` and the settings table), dev - tooling with no image consequence. +11. **The audit's deferred findings, done and waiting for the image.** All six + are fixed and host-proven, and ride the local image item 9's campaign + runs on. **K-4**: the SDMA script publishes end-of-data in a coherent + mailbox word before it signals (and clears the waypoint counter), and the + interrupt callback decodes on the mailbox, not on the host's arming, so an + end-of-data that arrives before an armed waypoint stops the run at once. + **K-8**: a resume's laser-off lead is the script's own inhibit mask, + applied to every GPIO word it writes and cleared at the waypoint byte; + run start restores the FIRE drive while the script is idle, the callback + writes nothing to the GPIO data register, and only the deceleration parks + the line. Both are proven on the host by the module's -Werror cross-build + and on the bench by the new `kernel.resume-lead` drill (end-of-data before + the waypoint at a 1 kHz tick, and a 1000-byte lead over FIRE bits with the + latch unlocked and the chain unarmed). **P-13**: the SDMA channel is + claimed through dmaengine (`dma_get_slave_channel`, released by + `dma_release_channel`, whose resource hooks hold the engine's clocks), and + the callback setter kills the tasklet before clearing and initializes it + only when setting. **P-14**: the inter-word wait states come from + `spi_transfer.word_delay`, which `pic.c` sets beside the post-transfer + `delay`. The kernel rebuilt clean with both patches. **B-16**: + `BB_SIGNATURE_LOCAL_DIRS_EXCLUDE` in the distro conf names `__pycache__` + and `.pytest_cache`, so a workstation's bytecode caches never enter a + file:// checksum (proven in the build VM: a cache under the package leaves + the fetch task alone, a source change reruns it). **FA-20**: the panel's + dev-server mock carries the daemon's tables and reply shapes, and a host + test in the forgectrl repository reads them out of the C sources and holds + the mock to them. The item closes with the campaign. **Deliberately not gated:** an armed GRBL job after an underrun cuts at the stale origin unless homing is required (GRBL mode permits unhomed cutting; the diff --git a/docs/CAMPAIGN-LOG.md b/docs/CAMPAIGN-LOG.md index b7fa4ba..ea8ebb0 100644 --- a/docs/CAMPAIGN-LOG.md +++ b/docs/CAMPAIGN-LOG.md @@ -7346,6 +7346,68 @@ Owed: the operator flashes the dev image, takes a fresh-boot baseline, and runs the full campaign (all three layer hashes moved), the campaign the release gate asks for. +## 2026-09-02: the audit's deferred findings, host-proven + +The operator's rule for the campaign set the order: no campaign until every +audit finding, the deferred six included, is on one image for a final test. +So the deferred batch was done at once, all of it local, all of it host-proven. + +**K-4 and K-8, in the SDMA script and the module.** The waypoint and the +end-of-data interrupts share one line, and the callback cannot fetch the +channel context (a channel-0 transfer, which sleeps), so it decoded on +host-side arming: an end-of-data before an armed waypoint read as the +waypoint. The script now writes 1 to a coherent mailbox word (allocated +before the dedicated pool is attached, so the pool stays the ring's alone; +its physical address rides a reserved context word) before its end-of-data +notify, clears the waypoint counter, and the callback decodes on the +mailbox; run start clears it. The resume lead moved into the script too: a +laser inhibit mask in a second reserved context word, ANDed out of every +GPIO word beside the motor lock, set by run start for an accelerating +forward run and cleared by the script at the waypoint byte. Run start +restores the FIRE drive while the script is idle, the callback writes +nothing to the GPIO data register, and only the deceleration parks the line +(the direction register alone). A resume with no lead plays laser-less for +the whole run, as before, now by construction. The script grew from 160 to +173 instructions; its branches are 8-bit displacements and the growth pushed +one past the range, which the assembler refuses, so the layout changed: the +waypoint action, the power-level path and the first end-of-data tick sit +past the main loop, reached by short branches; the worst displacement is ++120 of 127. Host proof: the module cross-compiles against the pinned +kernel with -Werror, its host tests pass. The bench proof is the new +`kernel.resume-lead` drill, two phases behind one takeover at a 1 kHz tick: +E, a resume whose lead is longer than the data, where a lost end-of-data +would show as 255 ms; L, a 1000-byte lead over FIRE bits with the latch +unlocked and the chain unarmed, the FIRE line sampled through the run. The +catalog counts 55 tests, 0 uncovered. + +**P-13 and P-14, in the kernel patches.** `sdma_get_channel()` claims the +channel through `dma_get_slave_channel()`, whose resource hook holds the +engine's ipg/ahb clocks the way the explicit enables did, and +`sdma_put_channel()` is `dma_release_channel()`; the callback setter kills +the tasklet before clearing and initializes it only when setting. The SPI +patch reads `spi_transfer.word_delay` for the PERIODREG wait states, and +`pic.c` sets `word_delay` beside `delay` (the post-transfer gap the factory +kernel also had). The hunks were edited in place; do_patch took them and +the kernel rebuilt clean. + +**B-16.** The real fix is bitbake's own knob: `BB_SIGNATURE_LOCAL_DIRS_EXCLUDE` +in the distro conf names `__pycache__` and `.pytest_cache` beside the VCS +directories, so the file fetcher's checksum never sees a bytecode cache. +The experiment in the build VM: a fetch on a clean tree ran the task; a +cache injected under the package directory left 2 of 2 tasks alone; a real +source change ran the task again. The CI's `-B` stays as a second layer. + +**FA-20.** The panel's dev-server mock carried 10 gate rows of the daemon's +22, lacked 18 settings keys, accepted a mode the daemon refuses, and sent +`/cool/status`, `/diag`, `/boot`, `/update` and `/slots` shapes the daemon +does not. It now carries the daemon's tables and reply shapes, and a host +test (`tests/test_devserver_mock.py`, 15 cases, one CI step) parses the C +tables and holds the mock to them. + +Owed: one local image with the batch (the module pinned from its local +commit, the rest from the pushed pins), the flash, the fresh-boot baseline +and the full campaign; then the push in CI order and the pin bumps. + ## Reference notes ### Head-IRQ source validation — the beam-emission hypothesis diff --git a/forgetest/forgetest/suite/kernel.py b/forgetest/forgetest/suite/kernel.py index 4437b06..64ae9f1 100644 --- a/forgetest/forgetest/suite/kernel.py +++ b/forgetest/forgetest/suite/kernel.py @@ -679,3 +679,130 @@ def fire_line(ctx): except OSError: pass ctx.log("safe state restored: state=%s latch=LOCKED", rd("cnc/state")) + + +# ---------------------------------------------------------------- resume lead + +@test("kernel.resume-lead", title="A resume's laser-off lead, and an end-of-data before the waypoint", + subsystem="kernel", kind="auto", hardware="takeover", est_min=2, + covers=_KERNEL_COVERS, + requires=["kernel.k1-k2"], precheck=hv_off, + steps=["Phase L unlocks the latch with a zero-duty stream, so the drill starts only while " + "the HV supply does not report good (true at idle; if it is refused, open the lid - " + "the safety chain holds HV off - and start it again)."], + description="Two phases behind one takeover, zero duty, motors locked, a 1 kHz tick. E: a " + "resume whose lead is longer than the data (the ring drains before the waypoint) " + "ends the run at end-of-data within a few ticks, not one re-notify period (255 " + "ticks) later: the script publishes end-of-data in its mailbox and the host " + "decodes on that, not on the waypoint it was still waiting for. L: with the " + "latch unlocked and the chain unarmed, a resume with a 1000-byte lead over a " + "stream of FIRE bits keeps the FIRE line low through the lead and drives it from " + "the waypoint byte on. The script masks the laser bits itself, so no host write " + "reaches the GPIO data register while the script runs; LASER_ON stays off (the " + "safety AND-gate holds).") +def resume_lead(ctx): + ev = ctx.evidence + require_hv_off(ctx) + tick = 1000 + with ctx.takeover(): + try: + # ---- E: end-of-data before the waypoint + stream = POWER0 + PAD * (tick // 2) # 0.5 s of pads + expected = len(stream) / tick + ctx.log("E: %d bytes = %.1f s of pads at %d Hz; resume with a %d-byte lead (never reached)", + len(stream), expected, tick, 10 * tick) + wr("cnc/motor_lock", 15) + wr("cnc/laser_latch", 1) + wr("cnc/ramp_rate", 125000) + wr("cnc/step_freq", tick) + snap(ctx, "E pre") + with PulseDevice(ctx) as dev: + dev.rewind() + wr("cnc/enable", 1) + ctx.sleep(0.5) + dev.write(stream) + underruns_before = int(rd("cnc/underruns")) + t0 = time.time() + wr("cnc/resume", 10 * tick) + state = wait_state(ctx, "running", 2, poll=0.002) + ctx.check(state == "running", "E: the resume did not start (state=%s)", state) + state = wait_state(ctx, "idle", 5, poll=0.002) + dt = time.time() - t0 + ev["E"] = {"run_to_idle_s": round(dt, 3), "data_s": expected, "state": state, + "underruns_before": underruns_before, + "underruns_after": int(rd("cnc/underruns")), "faults": rd("cnc/faults")} + ctx.log("E: %s after %.3f s (the data is %.3f s; a lost end-of-data would add %.3f s)", + state, dt, expected, 255.0 / tick) + ctx.check(state == "idle", "E: state %s after the data ran out", state) + ctx.check(dt < expected + 0.15, + "E: the run ended %.3f s after its data: end-of-data decoded late", dt - expected) + ctx.check(ev["E"]["underruns_after"] == underruns_before, + "E: an underrun was counted on a normal completion") + ctx.check(ev["E"]["faults"] == "0", "E: faults=%s", ev["E"]["faults"]) + ctx.log("E PASS: end-of-data before the waypoint ended the run on time") + + # ---- L: the lead, latch unlocked, chain unarmed + check_hv_off(ctx) + lead = tick # 1.0 s of FIRE bits held off + lead_s = lead / tick + stream = POWER0 + FIRE * (2 * tick) + PAD * (tick // 5) + ctx.log("L: %d bytes = %.1f s (%.1f s of FIRE bits then pads); lead %d bytes = %.1f s; " + "latch UNLOCKED, chain unarmed", len(stream), len(stream) / tick, + 2 * tick / tick, lead, lead_s) + wr("cnc/motor_lock", 15) + wr("cnc/laser_latch", 1) + wr("cnc/step_freq", tick) + snap(ctx, "L pre") + samples = [] + state = "" + with PulseDevice(ctx) as dev: + dev.rewind() + wr("cnc/enable", 1) + ctx.sleep(0.5) + dev.write(stream) + try: + wr("cnc/laser_latch", 0) + t0 = time.time() + wr("cnc/resume", lead) + n = 0 + while time.time() - t0 < 10: + en, on = rd("cnc/laser_enable"), rd("cnc/laser_on") + samples.append((round(time.time() - t0, 4), en, on)) + state = rd("cnc/state") + if state != "running" and time.time() - t0 > 0.2: + break + n += 1 + if n % 200 == 0: + ctx.checkpoint() + finally: + wr("cnc/laser_latch", 1) + snap(ctx, "L post") + driven = [s for s in samples if s[1] != "0"] + first = driven[0][0] if driven else None + last = driven[-1][0] if driven else None + inside_lead = [s for s in driven if s[0] < lead_s - 0.02] + ev["L"] = {"samples": len(samples), "driven_samples": len(driven), + "first_driven_s": first, "last_driven_s": last, "end_state": state, + "any_laser_on": any(s[2] != "0" for s in samples), + "inside_lead": inside_lead[:5], "laser_on_sampled": rd("cnc/laser_on_sampled"), + "faults": rd("cnc/faults")} + ctx.log("L: %d samples, FIRE driven in %d, first at %s s, last at %s s, end state %s", + len(samples), len(driven), first, last, state) + ctx.check(state == "idle", "L: ended in %s", state) + ctx.check(not inside_lead, "L: FIRE driven inside the lead: %s", inside_lead[:5]) + ctx.check(driven, "L: FIRE never driven after the lead (the inhibit never cleared)") + ctx.check(first is not None and first < lead_s + 0.3, + "L: FIRE first driven at %s s; the lead ends at %.1f s", first, lead_s) + ctx.check(not ev["L"]["any_laser_on"], + "L: LASER_ON active with the chain unarmed - the AND-gate did not hold") + ctx.check(rd("cnc/laser_enable") == "0", "L: FIRE still driven after end-of-data") + ctx.log("L PASS: FIRE low through the %.1f s lead, driven from the waypoint on, LASER_ON off", + lead_s) + finally: + wr("cnc/laser_latch", 1) + try: + wr("cnc/ramp_rate", 125000) + wr("cnc/disable", 1) # the script's safe state + except OSError: + pass + ctx.log("safe state restored: state=%s latch=LOCKED", rd("cnc/state")) diff --git a/meta-forgefirm/conf/distro/forgefirm.conf b/meta-forgefirm/conf/distro/forgefirm.conf index cff3e54..3f2728a 100644 --- a/meta-forgefirm/conf/distro/forgefirm.conf +++ b/meta-forgefirm/conf/distro/forgefirm.conf @@ -28,3 +28,8 @@ BAD_RECOMMENDATIONS += "eudev-hwdb" # signing modules (rootfs must fit the 200 MiB factory slot). VIRTUAL-RUNTIME_base-utils-syslog = "rsyslog" PACKAGECONFIG:pn-rsyslog = "rsyslogd rsyslogrt klog inet regexp" + +# Local (file://) source checksums leave out a workstation's Python bytecode +# caches: the forgetest recipe fetches its package directory whole, and a +# host test run must not move a task hash without a source change. +BB_SIGNATURE_LOCAL_DIRS_EXCLUDE = "CVS .bzr .git .hg .osc .p4 .repo .svn __pycache__ .pytest_cache"