mirror of
https://github.com/openglow-org/forgefirm.git
synced 2026-09-27 16:51:12 -07:00
docs: the pooled bench session's first pass, and the PIC read pattern
CAMPAIGN-LOG: the unattended set on image 20260902144848, the six harness and diagnostic defects it found and their fixes, the numbers, and the board's state at the end of the pass. BRINGUP: item 9 names the reset in the hang case and the session under way; the facts bank gains the PIC read-pattern measurement; Next work item 10 is a pacing of PIC reads in the kernel.
This commit is contained in:
+28
-1
@@ -1004,6 +1004,19 @@ is committed.
|
||||
share off. With the pump on, a heater slug reaches the upstream sensor
|
||||
within seconds and inflates the instant reading by a degree; the warm-up
|
||||
release therefore judges a one-minute rolling minimum of that reading.
|
||||
- **Coolant-ADC readings depend on the read pattern.** What the PIC returns
|
||||
for a thermistor depends on how soon the read follows the previous PIC read
|
||||
(measured 2026-09-02 on the two coolant channels): the second of a pair
|
||||
issued within 0.1 ms comes back 6 to 8 counts high with a wide spread,
|
||||
either sensor, either order; a pair 0.5 to 10 ms apart reads tight and a
|
||||
steady 3 counts (about 0.2 C) above sparse reads; and concurrent readers
|
||||
land such pairs at random, so a fast reader sees excursions of 10 to 15
|
||||
counts in a share of its samples that grows with the read rate (a third at
|
||||
100 Hz). The `aa-offset-calibrate` diagnostic reads its two sensors 31 ms
|
||||
apart and reduces each window to an interquartile mean; the cooling engine
|
||||
and `/status` still read the PIC back to back (the bias is inside the
|
||||
gates' margins). A pacing of PIC reads in the kernel would give every reader
|
||||
the same value (Next work).
|
||||
- **Coolant-ADC offsets around a lit tube.** The air-assist fan's return
|
||||
current rides a ground path the thermistor reference shares, so both coolant
|
||||
sensors read about 1.2 C low at the run duty (proportional to the fan's
|
||||
@@ -1340,8 +1353,13 @@ pump flag. Beam detect stays with item 6.
|
||||
campaign is a full one; the drills that prove the safety fixes directly are
|
||||
`kernel.deadman-close` (new), `cloud.lid-during-button-wait` (now a job
|
||||
longer than the ring), `cloud.verdict-hold` (new, about twelve minutes with
|
||||
the loop heater), `motion.deadman` (the hang case now recovers on `$X`),
|
||||
the loop heater), `motion.deadman` (the hang case recovers on a soft
|
||||
reset and `$X`: the stream fault is a critical alarm, which the core
|
||||
unlocks only after a reset, and the reset is what re-arms the stream),
|
||||
and the `cooling.*` set); then the push in CI order and the pin bumps.
|
||||
The session runs on image 20260902144848; the harness and diagnostic
|
||||
defects its first pass found are fixed in local commits (the dated record
|
||||
is in CAMPAIGN-LOG).
|
||||
Decisions taken in the remediation that the operator confirms or reverses:
|
||||
the release image has no shell login (the install page now says so); the
|
||||
cloud client holds and resumes on the cooling verdict with a 30-minute
|
||||
@@ -1355,3 +1373,12 @@ pump flag. Beam detect stays with item 6.
|
||||
that floats high gets a pull-down pad in the device tree); one forced
|
||||
kernel hard hang to confirm that the bootloader's watchdog-timeout path
|
||||
boots the factory recovery and that a power cycle returns from it.
|
||||
|
||||
10. **PIC read pacing.** A PIC read within a fraction of a millisecond of
|
||||
the previous one returns a disturbed value (facts bank, "Coolant-ADC
|
||||
readings depend on the read pattern"). Serialize the PIC transactions in
|
||||
the kernel module with a minimum spacing (a millisecond is enough by the
|
||||
measurement), so every reader, the engine, `/status`, the diagnostics and
|
||||
a bench sampler, sees the same value whatever the others do. A module
|
||||
change: it rides the next image flash, with the coolant-reading tests
|
||||
(`cooling.aa-offset-calibrate`, `cooling.flow-verify`) as its proof.
|
||||
|
||||
@@ -7136,6 +7136,89 @@ bench measurements, one local image build of both images, then the pushes
|
||||
in CI order (forgefirm first), the pin bumps, and the operator's
|
||||
decisions listed in the same item.
|
||||
|
||||
## 2026-09-02: the pooled bench session, first pass: the unattended set
|
||||
|
||||
Image 20260902144848 (dev), built locally from the audit remediation, was
|
||||
flashed to the SD slot; the fresh-boot reference was taken at uptime 24 s.
|
||||
The unattended queue ran nine times. Every stop was a harness or
|
||||
diagnostic defect, none an image defect, and every fix went the same way:
|
||||
host tests, hot deploy to the board, bench PASS, one local commit. At the
|
||||
end the unattended set is green and the campaign is open with 43 of 54
|
||||
tests satisfied; the 11 left are the attended queue, the operator's
|
||||
(`laser.emission-witness`, `cooling.flow-under-load`, `laser.m5-rapid-dark`,
|
||||
`laser.disarm-in-hold`, `laser.armed-kill`, `laser.pause-resume-lid-cancel`,
|
||||
`cloud.service-protocol`, `cloud.lid-interlock-abort`, `cloud.pause-resume`,
|
||||
`cloud.oversize-stream`, `cloud.paused-lid-cancel`), plus the two bench
|
||||
measurements BRINGUP item 9 lists.
|
||||
|
||||
The defects, in the order the queue found them:
|
||||
|
||||
1. `image.health` compared the running kernel's full release string with
|
||||
the manifest's modules directory, which the remediation (forgefirm
|
||||
88ec984) lists without the `LOCALVERSION_AUTO` hash. The test strips the
|
||||
same suffix from both sides (forgefirm 133b61a).
|
||||
2. `kernel.fire-line` phase B was refused by the HV-off latch-unlock gate
|
||||
(forgefirm 64f552fc, its first bench run): the gate ran within a second
|
||||
of phase A's run, and the one-shot holds `CHARGE_PUMP_ALIVE` for 0.45 s
|
||||
after the last 200 ms feed. The gate now waits up to 3 s for the chain
|
||||
to release and records the wait; the chain released after 0.41 s at each
|
||||
of the three phase boundaries (forgefirm b3efab9).
|
||||
3. `motion.deadman`'s hang case (new in the remediation) sent `$X` alone.
|
||||
The stream fault raises Alarm 17, a critical event: the core refuses
|
||||
`$X` with error 79 until a soft reset, and the reset is what re-arms the
|
||||
stream (grblHAL 38b450e). The drill records the refusal, resets,
|
||||
unlocks, requires the ring back at its idle free count, then jogs; the
|
||||
final assertion had also read the state off the report dict as a string
|
||||
(forgefirm d536963). Bench: kill respawn 1.2 s, hang to underrun 0.21 s,
|
||||
`$X` -> `ALARM:17 error:79`, reset and `$X` -> Idle, ring 33521664 of
|
||||
33521664, jog Jog -> Idle, restart retook supervision.
|
||||
4. `cooling.aa-offset-calibrate` refused four runs (spreads 17.9, 11.3,
|
||||
23.2 and 8.3 counts over six edges of a 15-count step). Two causes. The
|
||||
queue ran it right after `cooling.flow-verify`, whose no-flow trial
|
||||
heats the tube water by 17 C; the warm slug circulates past the sensors
|
||||
for minutes and the stationary gate, which compares split-half means,
|
||||
passes at a wave's crest: the calibration is registered before the
|
||||
heater tools now (forgefirm 1fef9c6). And the readings themselves: a
|
||||
PIC read's value depends on how soon it follows the previous PIC read
|
||||
(a pair 0.1 ms apart: the second reads 6 to 8 counts high with a wide
|
||||
spread; 0.5 to 10 ms apart: tight, a steady 3 counts above sparse
|
||||
reads; other readers land such pairs at random). The tool read both
|
||||
sensors back to back at 8 Hz and averaged; an 8 Hz sampler with spaced
|
||||
reads found every edge within 2 counts of 15 at the same moments. The
|
||||
tool reads the sensors 31 ms apart at 16 Hz, reduces each window to its
|
||||
interquartile mean, logs every window's count, extremes and value, and
|
||||
its spread limit is 12 (forgectrl 0b35f8a; docs 7df312b). Bench: spread
|
||||
3.6 and 4.2 standalone, 7.9 under the queue, 5.0 on the final binary;
|
||||
the idle windows within 0.6 counts across a run. The cooling engine and
|
||||
`/status` still read the PIC back to back; the bias is inside the gates'
|
||||
margins and is a facts-bank entry and Next work item 10 (a pacing of PIC
|
||||
reads in the kernel).
|
||||
5. `update.slots-and-signature`'s apply section (forgefirm b182a5a, never
|
||||
bench-run) required 200 where the daemon answers 202 with `started`,
|
||||
and looked for "not signed", which is the daemon's wording ("archive is
|
||||
not signed with the ForgeFIRM release key"); its cleanup had deleted the
|
||||
staged archive under the running job, which is why the first run's job
|
||||
ended with "not a usable fwup archive" (forgefirm 7605a90).
|
||||
6. A queue started 4 s after a forgetest restart ran 7 tests instead of
|
||||
10: the bench page's `/state` poll had a fixture probe in flight (an
|
||||
mDNS answer), the probe stamped its time at its start, and the queue
|
||||
start read the stale "no fixture", so the three operator tests the
|
||||
fixture runs (`cloud.mode-switch`, `cloud.lid-during-button-wait`,
|
||||
`cloud.verdict-hold`) were routed to nobody. The probe is serialized
|
||||
and stamped at completion; `tests/test_fixture.py` holds the race
|
||||
(forgefirm 7605a90).
|
||||
|
||||
The campaign rules cost what they promise: a FAIL closes the campaign, so
|
||||
the always-required core (`image.health` and the five `kernel.*` drills,
|
||||
about five minutes) ran again after every stop, nine times in all.
|
||||
|
||||
The board at the end of the pass: the image's forgectrl is replaced by the
|
||||
0b35f8a build, and the forgetest suite files `image.py`, `kernel.py`,
|
||||
`motion.py`, `cooling.py`, `update.py` and `runner.py` are the committed
|
||||
ones, all hot-deployed; the manifest still names the pins the image was
|
||||
built from until the next flash. `/tmp` is empty and `/data` holds nothing
|
||||
of the session's. The head was returned to its start by the baselines.
|
||||
|
||||
## Reference notes
|
||||
|
||||
### Head-IRQ source validation — the beam-emission hypothesis
|
||||
|
||||
Reference in New Issue
Block a user