59 lines
2.8 KiB
Markdown
59 lines
2.8 KiB
Markdown
# The browser suite is flaky under load — UNRESOLVED, and owned by design-dev
|
||
|
||
_2026-09-23 · booth_
|
||
|
||
⚠ **OPEN. Not fixed. Do not read a green suite as proof of anything without
|
||
re-running it.**
|
||
|
||
## What is observed, with its limits
|
||
|
||
**Three different browser tests** have each failed once under full-suite load
|
||
while passing repeatedly in isolation:
|
||
|
||
| test | owner | isolation | full suite |
|
||
|---|---|---|---|
|
||
| `test_the_keyboard_flag_actually_submits` | booth-dev | 5/5 pass | 1 failure |
|
||
| `test_a_failed_save_says_so_reloads_and_never_re_posts` | design-dev | 3/3 pass | 1 failure |
|
||
| `test_the_review_keys_judge_in_place_and_stay_out_of_the_note` | design-dev | 10/10 + 5/5 pass | 1 in ~9 |
|
||
|
||
Three different tests points at **the environment under contention**, not at any
|
||
one test. That is a direction, not a finding.
|
||
|
||
## Two real defects found chasing it — NEITHER PROVEN TO BE THE CAUSE
|
||
|
||
1. **A keypress race.** The flag test fired `ArrowRight` and `f` back to back,
|
||
assuming the first had finished — but `focus()` does a `scrollIntoView`, so
|
||
under load `f` could arrive with no cursor set. Now waits for
|
||
`figure.item.is-cursor`.
|
||
2. **A port TOCTOU in BOTH browser fixtures.** Each did `bind → getsockname →
|
||
CLOSE → hand uvicorn the port NUMBER`, leaving a window for the kernel to
|
||
give that port away — and this suite runs two browser files that each start a
|
||
server per test, so the competitor is the other file. The bound socket is now
|
||
passed to `server.run(sockets=[sock])`.
|
||
|
||
**Since those fixes: one failure in three full runs. n=3 CANNOT distinguish
|
||
that from the prior rate, and no claim of improvement is made.**
|
||
|
||
## Who owns it and what the method is
|
||
|
||
**The operator ruled: "let him diagnose it properly."** design-dev owns it. His
|
||
method: a trace hook keeping a Playwright trace (screenshots + DOM snapshots)
|
||
for every browser test that fails, captured **from the run that fails**, then
|
||
full-suite runs until red and the artefact decides. Test-only infra, on his
|
||
branch as its own commit.
|
||
|
||
⚠ **The methodological trap, and why the artefact must come from the full
|
||
suite:** the failure only reproduces there. A narrowed repro that passes proves
|
||
nothing and will be mistaken for a fix.
|
||
|
||
## Addendum 2026-09-24 — the offline fix, and what 0/24 can and cannot say
|
||
|
||
design-dev's suspect was Google Fonts stalling "networkidle". A stalled font
|
||
request reproduces the exact error (which shows sufficiency only). The fix
|
||
landed in `b92b002`: the test browser has no internet, with a positive control
|
||
per fixture. Traced runs: 0/15 (light) and 0/8 (heavy). No trace ever caught
|
||
the stalled request. Untraced after the fix: **0 reds in 24**, against a pre-fix
|
||
rate of 1 in 8 that is itself one red in eight runs (95% CI roughly 0.3–53%).
|
||
At a true rate of 1 in 20, 0/24 happens 29% of the time. So it's consistent
|
||
with the fix and not a confirmation of the cause.
|