mirror of
https://github.com/openglow-org/forgefirm.git
synced 2026-09-28 01:01:12 -07:00
docs: the pooled bench session's first pass, and the PIC read pattern
CAMPAIGN-LOG: the unattended set on image 20260902144848, the six harness and diagnostic defects it found and their fixes, the numbers, and the board's state at the end of the pass. BRINGUP: item 9 names the reset in the hang case and the session under way; the facts bank gains the PIC read-pattern measurement; Next work item 10 is a pacing of PIC reads in the kernel.
This commit is contained in:
+28
-1
@@ -1004,6 +1004,19 @@ is committed.
|
|||||||
share off. With the pump on, a heater slug reaches the upstream sensor
|
share off. With the pump on, a heater slug reaches the upstream sensor
|
||||||
within seconds and inflates the instant reading by a degree; the warm-up
|
within seconds and inflates the instant reading by a degree; the warm-up
|
||||||
release therefore judges a one-minute rolling minimum of that reading.
|
release therefore judges a one-minute rolling minimum of that reading.
|
||||||
|
- **Coolant-ADC readings depend on the read pattern.** What the PIC returns
|
||||||
|
for a thermistor depends on how soon the read follows the previous PIC read
|
||||||
|
(measured 2026-09-02 on the two coolant channels): the second of a pair
|
||||||
|
issued within 0.1 ms comes back 6 to 8 counts high with a wide spread,
|
||||||
|
either sensor, either order; a pair 0.5 to 10 ms apart reads tight and a
|
||||||
|
steady 3 counts (about 0.2 C) above sparse reads; and concurrent readers
|
||||||
|
land such pairs at random, so a fast reader sees excursions of 10 to 15
|
||||||
|
counts in a share of its samples that grows with the read rate (a third at
|
||||||
|
100 Hz). The `aa-offset-calibrate` diagnostic reads its two sensors 31 ms
|
||||||
|
apart and reduces each window to an interquartile mean; the cooling engine
|
||||||
|
and `/status` still read the PIC back to back (the bias is inside the
|
||||||
|
gates' margins). A pacing of PIC reads in the kernel would give every reader
|
||||||
|
the same value (Next work).
|
||||||
- **Coolant-ADC offsets around a lit tube.** The air-assist fan's return
|
- **Coolant-ADC offsets around a lit tube.** The air-assist fan's return
|
||||||
current rides a ground path the thermistor reference shares, so both coolant
|
current rides a ground path the thermistor reference shares, so both coolant
|
||||||
sensors read about 1.2 C low at the run duty (proportional to the fan's
|
sensors read about 1.2 C low at the run duty (proportional to the fan's
|
||||||
@@ -1340,8 +1353,13 @@ pump flag. Beam detect stays with item 6.
|
|||||||
campaign is a full one; the drills that prove the safety fixes directly are
|
campaign is a full one; the drills that prove the safety fixes directly are
|
||||||
`kernel.deadman-close` (new), `cloud.lid-during-button-wait` (now a job
|
`kernel.deadman-close` (new), `cloud.lid-during-button-wait` (now a job
|
||||||
longer than the ring), `cloud.verdict-hold` (new, about twelve minutes with
|
longer than the ring), `cloud.verdict-hold` (new, about twelve minutes with
|
||||||
the loop heater), `motion.deadman` (the hang case now recovers on `$X`),
|
the loop heater), `motion.deadman` (the hang case recovers on a soft
|
||||||
|
reset and `$X`: the stream fault is a critical alarm, which the core
|
||||||
|
unlocks only after a reset, and the reset is what re-arms the stream),
|
||||||
and the `cooling.*` set); then the push in CI order and the pin bumps.
|
and the `cooling.*` set); then the push in CI order and the pin bumps.
|
||||||
|
The session runs on image 20260902144848; the harness and diagnostic
|
||||||
|
defects its first pass found are fixed in local commits (the dated record
|
||||||
|
is in CAMPAIGN-LOG).
|
||||||
Decisions taken in the remediation that the operator confirms or reverses:
|
Decisions taken in the remediation that the operator confirms or reverses:
|
||||||
the release image has no shell login (the install page now says so); the
|
the release image has no shell login (the install page now says so); the
|
||||||
cloud client holds and resumes on the cooling verdict with a 30-minute
|
cloud client holds and resumes on the cooling verdict with a 30-minute
|
||||||
@@ -1355,3 +1373,12 @@ pump flag. Beam detect stays with item 6.
|
|||||||
that floats high gets a pull-down pad in the device tree); one forced
|
that floats high gets a pull-down pad in the device tree); one forced
|
||||||
kernel hard hang to confirm that the bootloader's watchdog-timeout path
|
kernel hard hang to confirm that the bootloader's watchdog-timeout path
|
||||||
boots the factory recovery and that a power cycle returns from it.
|
boots the factory recovery and that a power cycle returns from it.
|
||||||
|
|
||||||
|
10. **PIC read pacing.** A PIC read within a fraction of a millisecond of
|
||||||
|
the previous one returns a disturbed value (facts bank, "Coolant-ADC
|
||||||
|
readings depend on the read pattern"). Serialize the PIC transactions in
|
||||||
|
the kernel module with a minimum spacing (a millisecond is enough by the
|
||||||
|
measurement), so every reader, the engine, `/status`, the diagnostics and
|
||||||
|
a bench sampler, sees the same value whatever the others do. A module
|
||||||
|
change: it rides the next image flash, with the coolant-reading tests
|
||||||
|
(`cooling.aa-offset-calibrate`, `cooling.flow-verify`) as its proof.
|
||||||
|
|||||||
@@ -7136,6 +7136,89 @@ bench measurements, one local image build of both images, then the pushes
|
|||||||
in CI order (forgefirm first), the pin bumps, and the operator's
|
in CI order (forgefirm first), the pin bumps, and the operator's
|
||||||
decisions listed in the same item.
|
decisions listed in the same item.
|
||||||
|
|
||||||
|
## 2026-09-02: the pooled bench session, first pass: the unattended set
|
||||||
|
|
||||||
|
Image 20260902144848 (dev), built locally from the audit remediation, was
|
||||||
|
flashed to the SD slot; the fresh-boot reference was taken at uptime 24 s.
|
||||||
|
The unattended queue ran nine times. Every stop was a harness or
|
||||||
|
diagnostic defect, none an image defect, and every fix went the same way:
|
||||||
|
host tests, hot deploy to the board, bench PASS, one local commit. At the
|
||||||
|
end the unattended set is green and the campaign is open with 43 of 54
|
||||||
|
tests satisfied; the 11 left are the attended queue, the operator's
|
||||||
|
(`laser.emission-witness`, `cooling.flow-under-load`, `laser.m5-rapid-dark`,
|
||||||
|
`laser.disarm-in-hold`, `laser.armed-kill`, `laser.pause-resume-lid-cancel`,
|
||||||
|
`cloud.service-protocol`, `cloud.lid-interlock-abort`, `cloud.pause-resume`,
|
||||||
|
`cloud.oversize-stream`, `cloud.paused-lid-cancel`), plus the two bench
|
||||||
|
measurements BRINGUP item 9 lists.
|
||||||
|
|
||||||
|
The defects, in the order the queue found them:
|
||||||
|
|
||||||
|
1. `image.health` compared the running kernel's full release string with
|
||||||
|
the manifest's modules directory, which the remediation (forgefirm
|
||||||
|
88ec984) lists without the `LOCALVERSION_AUTO` hash. The test strips the
|
||||||
|
same suffix from both sides (forgefirm 133b61a).
|
||||||
|
2. `kernel.fire-line` phase B was refused by the HV-off latch-unlock gate
|
||||||
|
(forgefirm 64f552fc, its first bench run): the gate ran within a second
|
||||||
|
of phase A's run, and the one-shot holds `CHARGE_PUMP_ALIVE` for 0.45 s
|
||||||
|
after the last 200 ms feed. The gate now waits up to 3 s for the chain
|
||||||
|
to release and records the wait; the chain released after 0.41 s at each
|
||||||
|
of the three phase boundaries (forgefirm b3efab9).
|
||||||
|
3. `motion.deadman`'s hang case (new in the remediation) sent `$X` alone.
|
||||||
|
The stream fault raises Alarm 17, a critical event: the core refuses
|
||||||
|
`$X` with error 79 until a soft reset, and the reset is what re-arms the
|
||||||
|
stream (grblHAL 38b450e). The drill records the refusal, resets,
|
||||||
|
unlocks, requires the ring back at its idle free count, then jogs; the
|
||||||
|
final assertion had also read the state off the report dict as a string
|
||||||
|
(forgefirm d536963). Bench: kill respawn 1.2 s, hang to underrun 0.21 s,
|
||||||
|
`$X` -> `ALARM:17 error:79`, reset and `$X` -> Idle, ring 33521664 of
|
||||||
|
33521664, jog Jog -> Idle, restart retook supervision.
|
||||||
|
4. `cooling.aa-offset-calibrate` refused four runs (spreads 17.9, 11.3,
|
||||||
|
23.2 and 8.3 counts over six edges of a 15-count step). Two causes. The
|
||||||
|
queue ran it right after `cooling.flow-verify`, whose no-flow trial
|
||||||
|
heats the tube water by 17 C; the warm slug circulates past the sensors
|
||||||
|
for minutes and the stationary gate, which compares split-half means,
|
||||||
|
passes at a wave's crest: the calibration is registered before the
|
||||||
|
heater tools now (forgefirm 1fef9c6). And the readings themselves: a
|
||||||
|
PIC read's value depends on how soon it follows the previous PIC read
|
||||||
|
(a pair 0.1 ms apart: the second reads 6 to 8 counts high with a wide
|
||||||
|
spread; 0.5 to 10 ms apart: tight, a steady 3 counts above sparse
|
||||||
|
reads; other readers land such pairs at random). The tool read both
|
||||||
|
sensors back to back at 8 Hz and averaged; an 8 Hz sampler with spaced
|
||||||
|
reads found every edge within 2 counts of 15 at the same moments. The
|
||||||
|
tool reads the sensors 31 ms apart at 16 Hz, reduces each window to its
|
||||||
|
interquartile mean, logs every window's count, extremes and value, and
|
||||||
|
its spread limit is 12 (forgectrl 0b35f8a; docs 7df312b). Bench: spread
|
||||||
|
3.6 and 4.2 standalone, 7.9 under the queue, 5.0 on the final binary;
|
||||||
|
the idle windows within 0.6 counts across a run. The cooling engine and
|
||||||
|
`/status` still read the PIC back to back; the bias is inside the gates'
|
||||||
|
margins and is a facts-bank entry and Next work item 10 (a pacing of PIC
|
||||||
|
reads in the kernel).
|
||||||
|
5. `update.slots-and-signature`'s apply section (forgefirm b182a5a, never
|
||||||
|
bench-run) required 200 where the daemon answers 202 with `started`,
|
||||||
|
and looked for "not signed", which is the daemon's wording ("archive is
|
||||||
|
not signed with the ForgeFIRM release key"); its cleanup had deleted the
|
||||||
|
staged archive under the running job, which is why the first run's job
|
||||||
|
ended with "not a usable fwup archive" (forgefirm 7605a90).
|
||||||
|
6. A queue started 4 s after a forgetest restart ran 7 tests instead of
|
||||||
|
10: the bench page's `/state` poll had a fixture probe in flight (an
|
||||||
|
mDNS answer), the probe stamped its time at its start, and the queue
|
||||||
|
start read the stale "no fixture", so the three operator tests the
|
||||||
|
fixture runs (`cloud.mode-switch`, `cloud.lid-during-button-wait`,
|
||||||
|
`cloud.verdict-hold`) were routed to nobody. The probe is serialized
|
||||||
|
and stamped at completion; `tests/test_fixture.py` holds the race
|
||||||
|
(forgefirm 7605a90).
|
||||||
|
|
||||||
|
The campaign rules cost what they promise: a FAIL closes the campaign, so
|
||||||
|
the always-required core (`image.health` and the five `kernel.*` drills,
|
||||||
|
about five minutes) ran again after every stop, nine times in all.
|
||||||
|
|
||||||
|
The board at the end of the pass: the image's forgectrl is replaced by the
|
||||||
|
0b35f8a build, and the forgetest suite files `image.py`, `kernel.py`,
|
||||||
|
`motion.py`, `cooling.py`, `update.py` and `runner.py` are the committed
|
||||||
|
ones, all hot-deployed; the manifest still names the pins the image was
|
||||||
|
built from until the next flash. `/tmp` is empty and `/data` holds nothing
|
||||||
|
of the session's. The head was returned to its start by the baselines.
|
||||||
|
|
||||||
## Reference notes
|
## Reference notes
|
||||||
|
|
||||||
### Head-IRQ source validation — the beam-emission hypothesis
|
### Head-IRQ source validation — the beam-emission hypothesis
|
||||||
|
|||||||
Reference in New Issue
Block a user