From 774ae52b61acb761bef14a82d4211fa354da0b3c Mon Sep 17 00:00:00 2001 From: ScottW514 Date: Wed, 2 Sep 2026 17:02:33 -0400 Subject: [PATCH] docs: the pooled bench session's first pass, and the PIC read pattern CAMPAIGN-LOG: the unattended set on image 20260902144848, the six harness and diagnostic defects it found and their fixes, the numbers, and the board's state at the end of the pass. BRINGUP: item 9 names the reset in the hang case and the session under way; the facts bank gains the PIC read-pattern measurement; Next work item 10 is a pacing of PIC reads in the kernel. --- docs/BRINGUP.md | 29 +++++++++++++++- docs/CAMPAIGN-LOG.md | 83 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 111 insertions(+), 1 deletion(-) diff --git a/docs/BRINGUP.md b/docs/BRINGUP.md index b21fc37..a5bace4 100644 --- a/docs/BRINGUP.md +++ b/docs/BRINGUP.md @@ -1004,6 +1004,19 @@ is committed. share off. With the pump on, a heater slug reaches the upstream sensor within seconds and inflates the instant reading by a degree; the warm-up release therefore judges a one-minute rolling minimum of that reading. +- **Coolant-ADC readings depend on the read pattern.** What the PIC returns + for a thermistor depends on how soon the read follows the previous PIC read + (measured 2026-09-02 on the two coolant channels): the second of a pair + issued within 0.1 ms comes back 6 to 8 counts high with a wide spread, + either sensor, either order; a pair 0.5 to 10 ms apart reads tight and a + steady 3 counts (about 0.2 C) above sparse reads; and concurrent readers + land such pairs at random, so a fast reader sees excursions of 10 to 15 + counts in a share of its samples that grows with the read rate (a third at + 100 Hz). The `aa-offset-calibrate` diagnostic reads its two sensors 31 ms + apart and reduces each window to an interquartile mean; the cooling engine + and `/status` still read the PIC back to back (the bias is inside the + gates' margins). A pacing of PIC reads in the kernel would give every reader + the same value (Next work). - **Coolant-ADC offsets around a lit tube.** The air-assist fan's return current rides a ground path the thermistor reference shares, so both coolant sensors read about 1.2 C low at the run duty (proportional to the fan's @@ -1340,8 +1353,13 @@ pump flag. Beam detect stays with item 6. campaign is a full one; the drills that prove the safety fixes directly are `kernel.deadman-close` (new), `cloud.lid-during-button-wait` (now a job longer than the ring), `cloud.verdict-hold` (new, about twelve minutes with - the loop heater), `motion.deadman` (the hang case now recovers on `$X`), + the loop heater), `motion.deadman` (the hang case recovers on a soft + reset and `$X`: the stream fault is a critical alarm, which the core + unlocks only after a reset, and the reset is what re-arms the stream), and the `cooling.*` set); then the push in CI order and the pin bumps. + The session runs on image 20260902144848; the harness and diagnostic + defects its first pass found are fixed in local commits (the dated record + is in CAMPAIGN-LOG). Decisions taken in the remediation that the operator confirms or reverses: the release image has no shell login (the install page now says so); the cloud client holds and resumes on the cooling verdict with a 30-minute @@ -1355,3 +1373,12 @@ pump flag. Beam detect stays with item 6. that floats high gets a pull-down pad in the device tree); one forced kernel hard hang to confirm that the bootloader's watchdog-timeout path boots the factory recovery and that a power cycle returns from it. + +10. **PIC read pacing.** A PIC read within a fraction of a millisecond of + the previous one returns a disturbed value (facts bank, "Coolant-ADC + readings depend on the read pattern"). Serialize the PIC transactions in + the kernel module with a minimum spacing (a millisecond is enough by the + measurement), so every reader, the engine, `/status`, the diagnostics and + a bench sampler, sees the same value whatever the others do. A module + change: it rides the next image flash, with the coolant-reading tests + (`cooling.aa-offset-calibrate`, `cooling.flow-verify`) as its proof. diff --git a/docs/CAMPAIGN-LOG.md b/docs/CAMPAIGN-LOG.md index 0a3263d..b5527d0 100644 --- a/docs/CAMPAIGN-LOG.md +++ b/docs/CAMPAIGN-LOG.md @@ -7136,6 +7136,89 @@ bench measurements, one local image build of both images, then the pushes in CI order (forgefirm first), the pin bumps, and the operator's decisions listed in the same item. +## 2026-09-02: the pooled bench session, first pass: the unattended set + +Image 20260902144848 (dev), built locally from the audit remediation, was +flashed to the SD slot; the fresh-boot reference was taken at uptime 24 s. +The unattended queue ran nine times. Every stop was a harness or +diagnostic defect, none an image defect, and every fix went the same way: +host tests, hot deploy to the board, bench PASS, one local commit. At the +end the unattended set is green and the campaign is open with 43 of 54 +tests satisfied; the 11 left are the attended queue, the operator's +(`laser.emission-witness`, `cooling.flow-under-load`, `laser.m5-rapid-dark`, +`laser.disarm-in-hold`, `laser.armed-kill`, `laser.pause-resume-lid-cancel`, +`cloud.service-protocol`, `cloud.lid-interlock-abort`, `cloud.pause-resume`, +`cloud.oversize-stream`, `cloud.paused-lid-cancel`), plus the two bench +measurements BRINGUP item 9 lists. + +The defects, in the order the queue found them: + +1. `image.health` compared the running kernel's full release string with + the manifest's modules directory, which the remediation (forgefirm + 88ec984) lists without the `LOCALVERSION_AUTO` hash. The test strips the + same suffix from both sides (forgefirm 133b61a). +2. `kernel.fire-line` phase B was refused by the HV-off latch-unlock gate + (forgefirm 64f552fc, its first bench run): the gate ran within a second + of phase A's run, and the one-shot holds `CHARGE_PUMP_ALIVE` for 0.45 s + after the last 200 ms feed. The gate now waits up to 3 s for the chain + to release and records the wait; the chain released after 0.41 s at each + of the three phase boundaries (forgefirm b3efab9). +3. `motion.deadman`'s hang case (new in the remediation) sent `$X` alone. + The stream fault raises Alarm 17, a critical event: the core refuses + `$X` with error 79 until a soft reset, and the reset is what re-arms the + stream (grblHAL 38b450e). The drill records the refusal, resets, + unlocks, requires the ring back at its idle free count, then jogs; the + final assertion had also read the state off the report dict as a string + (forgefirm d536963). Bench: kill respawn 1.2 s, hang to underrun 0.21 s, + `$X` -> `ALARM:17 error:79`, reset and `$X` -> Idle, ring 33521664 of + 33521664, jog Jog -> Idle, restart retook supervision. +4. `cooling.aa-offset-calibrate` refused four runs (spreads 17.9, 11.3, + 23.2 and 8.3 counts over six edges of a 15-count step). Two causes. The + queue ran it right after `cooling.flow-verify`, whose no-flow trial + heats the tube water by 17 C; the warm slug circulates past the sensors + for minutes and the stationary gate, which compares split-half means, + passes at a wave's crest: the calibration is registered before the + heater tools now (forgefirm 1fef9c6). And the readings themselves: a + PIC read's value depends on how soon it follows the previous PIC read + (a pair 0.1 ms apart: the second reads 6 to 8 counts high with a wide + spread; 0.5 to 10 ms apart: tight, a steady 3 counts above sparse + reads; other readers land such pairs at random). The tool read both + sensors back to back at 8 Hz and averaged; an 8 Hz sampler with spaced + reads found every edge within 2 counts of 15 at the same moments. The + tool reads the sensors 31 ms apart at 16 Hz, reduces each window to its + interquartile mean, logs every window's count, extremes and value, and + its spread limit is 12 (forgectrl 0b35f8a; docs 7df312b). Bench: spread + 3.6 and 4.2 standalone, 7.9 under the queue, 5.0 on the final binary; + the idle windows within 0.6 counts across a run. The cooling engine and + `/status` still read the PIC back to back; the bias is inside the gates' + margins and is a facts-bank entry and Next work item 10 (a pacing of PIC + reads in the kernel). +5. `update.slots-and-signature`'s apply section (forgefirm b182a5a, never + bench-run) required 200 where the daemon answers 202 with `started`, + and looked for "not signed", which is the daemon's wording ("archive is + not signed with the ForgeFIRM release key"); its cleanup had deleted the + staged archive under the running job, which is why the first run's job + ended with "not a usable fwup archive" (forgefirm 7605a90). +6. A queue started 4 s after a forgetest restart ran 7 tests instead of + 10: the bench page's `/state` poll had a fixture probe in flight (an + mDNS answer), the probe stamped its time at its start, and the queue + start read the stale "no fixture", so the three operator tests the + fixture runs (`cloud.mode-switch`, `cloud.lid-during-button-wait`, + `cloud.verdict-hold`) were routed to nobody. The probe is serialized + and stamped at completion; `tests/test_fixture.py` holds the race + (forgefirm 7605a90). + +The campaign rules cost what they promise: a FAIL closes the campaign, so +the always-required core (`image.health` and the five `kernel.*` drills, +about five minutes) ran again after every stop, nine times in all. + +The board at the end of the pass: the image's forgectrl is replaced by the +0b35f8a build, and the forgetest suite files `image.py`, `kernel.py`, +`motion.py`, `cooling.py`, `update.py` and `runner.py` are the committed +ones, all hot-deployed; the manifest still names the pins the image was +built from until the next flash. `/tmp` is empty and `/data` holds nothing +of the session's. The head was returned to its start by the baselines. + ## Reference notes ### Head-IRQ source validation — the beam-emission hypothesis