1.0 KiB
[2026-09-17] A unit splitter must choose by SIZE, not by count — the inherited rule silently produced 22,000-word "chapters".
⭐⭐ A unit splitter must choose by SIZE, not by count — the inherited rule silently produced 22,000-word "chapters". scripts/r49-corpus/split_units.py: a marker mode qualifies only if its median unit is inside [600, 12000] AND no unit holds half the work; among qualifying modes PRIORITY breaks the tie (contents > chapter-word > roman > bare-numeral > caps-title), and paragraph-block sections are the fallback for works with no divisions. ⭐ Both rules exist because a control caught them: scoring by "median closest to target" chose caps-title (6 units, one holding 97% of the book) over True at First Light's real 20 chapters, because a median cannot see that distribution and a max bound can. Positive control: 8/10 Hemingway works reproduce the shipped mode and count exactly. Negative control: 40,000 words with no blank lines → 1 unit, refuses to fabricate divisions. Commit 705fa3a.