esh-pfi-infrastructure/services/semif-serve/bench-jev-2026-09-30/tables.md at 475d6d6bcb3c22280f3c4d4dc03c6cb3d369e969

Files

T

vh 475d6d6bcb docs: Jev candidate bench vs SemIf (fv-ml1 GPU 3) — Intern-Decision-4B is the replacement if SemIf is displaced

Operator ask relayed by brokkr-smithy-dev. Positive control (SemIf 187/231,
hard 0.613) reproduced exactly; negative control and a 4-restart noise floor
(0 flips) measured. On our replaced-baseline sets no candidate beats SemIf-with-
rotations beyond the ~4-pt floor; Intern-Decision-4B native matches it at one
ordering, is better on Wyrd, fits 9.7/10.3 GB and is 1.5-2.3x faster. JevBench
rank does not transfer. Raw per-item data kept out of git.

2026-09-30 05:00:45 -07:00

25 KiB

Raw Blame History

Key table

system	fits 12 GiB? rest / peak MiB	JevBench all /231 · hard /111	pooled /259 single · rot	Wyrd /84 single · rot	Δ pooled vs SemIf single (95% CI)	Δ pooled vs SemIf rot (95% CI)	21 criteria, ms	16 × ~3,900 tok, ms
SemIf (Qwen3.5-4B), baseline	yes: 9242 (9242–9250) / 12918 (12886–12918)	187 · 68	217 · 232	71 · 71	baseline	baseline	131 (131–132)	505 (502–519)
Plumb-4B, native	yes: 12064 / 12064	207 (206–207) · 89 (88–89)	224 (224–226) · 227 (227–228)	65 · 67	+2.7 (-2.7..+7.9)	-1.9 (-5.4..+1.5)	294 (292–295)	3378 (3374–3384)
Plumb-4B, drop-in	yes: 9242 / 12918	206 · 88	221 · 224	66 · 67	+1.5 (-3.7..+6.6)	-3.1 (-6.8..+0.4)	131 (131–132)	506 (506–507)
Imajev-4B, native	yes: 10404 / 11056	199 · 80	235 · 236	68 · 69	+6.9 (+1.6..+12.2)	+1.5 (-2.5..+5.6)	1219 (1212–1230)	5125 (5054–5140)
Intern-Decision-4B, native	yes: 9736 / 10290	202 · 83	240 · 236	79 · 77	+8.9 (+4.7..+13.2)	+1.5 (-2.0..+5.0)	88 (88–89)	215 (213–216)
Intern-Decision-4B, drop-in	yes: 9242 / 12918	202 · 85	232 · 233	74 · 73	+5.8 (+2.7..+9.0)	+0.4 (-2.3..+3.1)	132	508 (500–516)
JevK5 v0.2, native	yes: 12064 / 12064	198 (198–199) · 81 (81–82)	229 (228–229) · 228	67 (66–67) · 67	+4.6 (-0.8..+9.9)	-1.5 (-4.9..+1.9)	291 (291–292)	3360 (3358–3368)
JevK5 v0.2, drop-in	yes: 9242 / 12918	200 · 82	232 · 237	72 · 74	+5.8 (+1.1..+10.7)	+1.9 (-1.1..+5.1)	132	518 (505–519)
JevK5 v0.3, native	yes: 12064 / 12064	202 · 86	227 · 226	66 · 65	+3.9 (-0.9..+8.5)	-2.3 (-6.0..+1.4)	291 (291–292)	3363 (3355–3370)
JevK5 v0.3, drop-in	yes: 9242 / 12918	201 · 85	227 · 227	69 · 66	+3.9 (-0.8..+8.2)	-1.9 (-5.5..+1.8)	133 (131–133)	518 (506–521)

JevBench v1.2.16 public items (positive control + candidate score)

system	repeats	easy /48	standard /72	hard /111	all /231	all	p50 ms
SemIf (Qwen3.5-4B), baseline	4	48	71	68	187	0.810	36
Plumb-4B, native	3	48	70	89 (88–89)	207 (206–207)	0.896 (0.892–0.896)	20
Plumb-4B, drop-in	3	48	70	88	206	0.892	36
Imajev-4B, native	4	48	71	80	199	0.861	62 (62–63)
Intern-Decision-4B, native	4	48	71	83	202	0.874	40
Intern-Decision-4B, drop-in	3	48	69	85	202	0.874	36
JevK5 v0.2, native	3	48	69	81 (81–82)	198 (198–199)	0.857 (0.857–0.861)	20 (20–21)
JevK5 v0.2, drop-in	3	48	70	82	200	0.866	36 (36–37)
JevK5 v0.3, native	3	48	68	86	202	0.874	20 (20–21)
JevK5 v0.3, drop-in	3	48	68	85	201	0.870	36

Replaced-baseline sets, single (caller order)

system	pooled /259	authored144	perturb108	Cicada w1 /31	Cicada w2 /31	Wyrd /84	Wyrd place2 /21	Wyrd exit /21
SemIf (Qwen3.5-4B), baseline	217	116	83	30	20	71	21	16
Plumb-4B, native	224 (224–226)	130 (130–132)	84	29	19	65	21	10
Plumb-4B, drop-in	221	129	88	26	20	66	21	12
Imajev-4B, native	235	138	102	29	22	68	20	13
Intern-Decision-4B, native	240	132	100	29	22	79	21	19
Intern-Decision-4B, drop-in	232	128	102	30	19	74	21	17
JevK5 v0.2, native	229 (228–229)	132	85 (85–86)	30	16	67 (66–67)	21	12 (11–12)
JevK5 v0.2, drop-in	232	130	92	30	17	72	21	17
JevK5 v0.3, native	227	135	101 (101–102)	26	17	66	21	12
JevK5 v0.3, drop-in	227	132	99	26	21	69	21	15

Replaced-baseline sets, rotations (n rotations averaged)

system	pooled /259	authored144	perturb108	Cicada w1 /31	Cicada w2 /31	Wyrd /84	Wyrd place2 /21	Wyrd exit /21
SemIf (Qwen3.5-4B), baseline	232	131	93	30	19	71	21	18
Plumb-4B, native	227 (227–228)	130 (130–131)	87	30	18 (17–18)	67	21	11
Plumb-4B, drop-in	224	129	92	28	18	67	21	13
Imajev-4B, native	236	138	101	29	23	69	20	13
Intern-Decision-4B, native	236	130	99	29	22	77	21	19
Intern-Decision-4B, drop-in	233	130	102	30	19	73	21	17
JevK5 v0.2, native	228	131	91 (90–91)	30	16	67	21	12
JevK5 v0.2, drop-in	237	133	93	30	16	74	21	20
JevK5 v0.3, native	226	135	101	26	16	65	20	12
JevK5 v0.3, drop-in	227	135	101	26	19	66	20	13

Wyrd as one request per turn (4 decisions over one state: SemIf /decide/shared, native multi-question request)

system	repeats	Wyrd /84	place /21	place2 /21	exit /21	exit2 /21
SemIf (Qwen3.5-4B), baseline	3	71	17	21	16	17
Plumb-4B, native	2	65	17	21	10	17
Plumb-4B, drop-in	3	66	17	21	12	16
Imajev-4B, native	3	68	19	20	13	16
Intern-Decision-4B, native	3	77	16	21	20	20
Intern-Decision-4B, drop-in	3	74	18	21	17	18
JevK5 v0.2, native	3	67 (66–67)	17	21	12 (11–12)	17
JevK5 v0.2, drop-in	3	73	18	21	18	16
JevK5 v0.3, native	3	66	18	21	12	15
JevK5 v0.3, drop-in	3	69	18	21	15	15

Null control (content-free state) and positive controls inside the spike sets, single ordering

system	Cicada w1 blind /31	Wyrd blind /84	Cicada w1 controls /5	Wyrd controls /28
SemIf (Qwen3.5-4B), baseline	16	60	5	21
Plumb-4B, native	15	50	5	20
Plumb-4B, drop-in	15	60	4	21
Imajev-4B, native	16	48	5	24
Intern-Decision-4B, native	16	60	5	27
Intern-Decision-4B, drop-in	16	60	5	23
JevK5 v0.2, native	15	53	5	22 (21–22)
JevK5 v0.2, drop-in	15	60	5	23
JevK5 v0.3, native	16	60	5	20
JevK5 v0.3, drop-in	16	60	5	22

Paired against SemIf (each row's majority top over the repeats; group bootstrap 95% CI; exact McNemar)

system	set	cond	SemIf	cand	fixed	broken	Δ pts	95% CI	p
Plumb-4B, native	pooled	single	217/259	224/259	23	16	+2.7	-2.7..+7.9	0.3368
Plumb-4B, native	pooled	rotations	232/259	227/259	10	15	-1.9	-5.4..+1.5	0.4244
Plumb-4B, native	authored144	single	116/144	130/144	22	8	+9.7	+1.4..+18.1	0.0161
Plumb-4B, native	authored144	rotations	131/144	130/144	7	8	-0.7	-5.6..+3.5	1.0
Plumb-4B, native	cicada-w1	single	30/31	29/31	0	1	-3.2	-9.7..+0.0	1.0
Plumb-4B, native	cicada-w1	rotations	30/31	30/31	0	0	+0.0	+0.0..+0.0	1.0
Plumb-4B, native	wyrd	single	71/84	65/84	1	7	-7.1	-13.1..-1.2	0.0703
Plumb-4B, native	wyrd	rotations	71/84	67/84	3	7	-4.8	-11.9..+2.4	0.3438
Plumb-4B, native	perturbations108	single	83/108	84/108	10	9	+0.9	-9.3..+11.1	1.0
Plumb-4B, native	perturbations108	rotations	93/108	87/108	4	10	-5.6	-15.7..+3.7	0.1796
Plumb-4B, native	cicada-w2	single	20/31	19/31	1	2	-3.2	-12.9..+6.5	1.0
Plumb-4B, native	cicada-w2	rotations	19/31	18/31	1	2	-3.2	-16.1..+6.5	1.0
Plumb-4B, native	pooled	cand-single-vs-semif-rotations	232/259	224/259	10	18	-3.1	-7.3..+0.8	0.1849
Plumb-4B, native	wyrd	cand-single-vs-semif-rotations	71/84	65/84	3	9	-7.1	-15.5..+1.2	0.146
Plumb-4B, drop-in	pooled	single	217/259	221/259	21	17	+1.5	-3.7..+6.6	0.6271
Plumb-4B, drop-in	pooled	rotations	232/259	224/259	7	15	-3.1	-6.8..+0.4	0.1338
Plumb-4B, drop-in	authored144	single	116/144	129/144	20	7	+9.0	+1.4..+16.7	0.0192
Plumb-4B, drop-in	authored144	rotations	131/144	129/144	5	7	-1.4	-6.2..+3.5	0.7744
Plumb-4B, drop-in	cicada-w1	single	30/31	26/31	0	4	-12.9	-25.8..-3.2	0.125
Plumb-4B, drop-in	cicada-w1	rotations	30/31	28/31	0	2	-6.5	-16.1..+0.0	0.5
Plumb-4B, drop-in	wyrd	single	71/84	66/84	1	6	-6.0	-11.9..+0.0	0.125
Plumb-4B, drop-in	wyrd	rotations	71/84	67/84	2	6	-4.8	-11.9..+2.4	0.2891
Plumb-4B, drop-in	perturbations108	single	83/108	88/108	12	7	+4.6	-5.6..+14.8	0.3593
Plumb-4B, drop-in	perturbations108	rotations	93/108	92/108	6	7	-0.9	-10.2..+7.4	1.0
Plumb-4B, drop-in	cicada-w2	single	20/31	20/31	1	1	+0.0	-9.7..+9.7	1.0
Plumb-4B, drop-in	cicada-w2	rotations	19/31	18/31	0	1	-3.2	-9.7..+0.0	1.0
Plumb-4B, drop-in	pooled	cand-single-vs-semif-rotations	232/259	221/259	8	19	-4.2	-8.3..-0.4	0.0522
Plumb-4B, drop-in	wyrd	cand-single-vs-semif-rotations	71/84	66/84	2	7	-6.0	-13.1..+1.2	0.1797
Imajev-4B, native	pooled	single	217/259	235/259	30	12	+6.9	+1.6..+12.2	0.0079
Imajev-4B, native	pooled	rotations	232/259	236/259	14	10	+1.5	-2.5..+5.6	0.5413
Imajev-4B, native	authored144	single	116/144	138/144	25	3	+15.3	+9.0..+21.5	0.0
Imajev-4B, native	authored144	rotations	131/144	138/144	8	1	+4.9	+0.7..+9.0	0.0391
Imajev-4B, native	cicada-w1	single	30/31	29/31	0	1	-3.2	-9.7..+0.0	1.0
Imajev-4B, native	cicada-w1	rotations	30/31	29/31	0	1	-3.2	-9.7..+0.0	1.0
Imajev-4B, native	wyrd	single	71/84	68/84	5	8	-3.6	-13.1..+6.0	0.5811
Imajev-4B, native	wyrd	rotations	71/84	69/84	6	8	-2.4	-11.9..+7.1	0.7905
Imajev-4B, native	perturbations108	single	83/108	102/108	20	1	+17.6	+8.3..+27.8	0.0
Imajev-4B, native	perturbations108	rotations	93/108	101/108	9	1	+7.4	+1.9..+13.9	0.0215
Imajev-4B, native	cicada-w2	single	20/31	22/31	3	1	+6.5	-6.5..+19.4	0.625
Imajev-4B, native	cicada-w2	rotations	19/31	23/31	4	0	+12.9	+3.2..+25.8	0.125
Imajev-4B, native	pooled	cand-single-vs-semif-rotations	232/259	235/259	15	12	+1.2	-3.2..+5.5	0.7011
Imajev-4B, native	wyrd	cand-single-vs-semif-rotations	71/84	68/84	6	9	-3.6	-14.3..+6.0	0.6072
Intern-Decision-4B, native	pooled	single	217/259	240/259	28	5	+8.9	+4.7..+13.2	0.0001
Intern-Decision-4B, native	pooled	rotations	232/259	236/259	12	8	+1.5	-2.0..+5.0	0.5034
Intern-Decision-4B, native	authored144	single	116/144	132/144	20	4	+11.1	+4.9..+18.1	0.0015
Intern-Decision-4B, native	authored144	rotations	131/144	130/144	5	6	-0.7	-5.6..+4.2	1.0
Intern-Decision-4B, native	cicada-w1	single	30/31	29/31	0	1	-3.2	-9.7..+0.0	1.0
Intern-Decision-4B, native	cicada-w1	rotations	30/31	29/31	0	1	-3.2	-9.7..+0.0	1.0
Intern-Decision-4B, native	wyrd	single	71/84	79/84	8	0	+9.5	+3.6..+16.7	0.0078
Intern-Decision-4B, native	wyrd	rotations	71/84	77/84	7	1	+7.1	+1.2..+13.1	0.0703
Intern-Decision-4B, native	perturbations108	single	83/108	100/108	20	3	+15.7	+4.6..+26.9	0.0005
Intern-Decision-4B, native	perturbations108	rotations	93/108	99/108	9	3	+5.6	-2.8..+13.0	0.146
Intern-Decision-4B, native	cicada-w2	single	20/31	22/31	2	0	+6.5	+0.0..+16.1	0.5
Intern-Decision-4B, native	cicada-w2	rotations	19/31	22/31	4	1	+9.7	-3.2..+22.6	0.375
Intern-Decision-4B, native	pooled	cand-single-vs-semif-rotations	232/259	240/259	14	6	+3.1	-0.4..+6.7	0.1153
Intern-Decision-4B, native	wyrd	cand-single-vs-semif-rotations	71/84	79/84	9	1	+9.5	+2.4..+16.7	0.0215
Intern-Decision-4B, drop-in	pooled	single	217/259	232/259	18	3	+5.8	+2.7..+9.0	0.0015
Intern-Decision-4B, drop-in	pooled	rotations	232/259	233/259	8	7	+0.4	-2.3..+3.1	1.0
Intern-Decision-4B, drop-in	authored144	single	116/144	128/144	15	3	+8.3	+3.5..+13.9	0.0075
Intern-Decision-4B, drop-in	authored144	rotations	131/144	130/144	5	6	-0.7	-4.9..+3.5	1.0
Intern-Decision-4B, drop-in	cicada-w1	single	30/31	30/31	0	0	+0.0	+0.0..+0.0	1.0
Intern-Decision-4B, drop-in	cicada-w1	rotations	30/31	30/31	0	0	+0.0	+0.0..+0.0	1.0
Intern-Decision-4B, drop-in	wyrd	single	71/84	74/84	3	0	+3.6	+0.0..+8.3	0.25
Intern-Decision-4B, drop-in	wyrd	rotations	71/84	73/84	3	1	+2.4	-2.4..+7.1	0.625
Intern-Decision-4B, drop-in	perturbations108	single	83/108	102/108	20	1	+17.6	+7.4..+28.7	0.0
Intern-Decision-4B, drop-in	perturbations108	rotations	93/108	102/108	9	0	+8.3	+2.8..+15.7	0.0039
Intern-Decision-4B, drop-in	cicada-w2	single	20/31	19/31	0	1	-3.2	-9.7..+0.0	1.0
Intern-Decision-4B, drop-in	cicada-w2	rotations	19/31	19/31	0	0	+0.0	+0.0..+0.0	1.0
Intern-Decision-4B, drop-in	pooled	cand-single-vs-semif-rotations	232/259	232/259	10	10	+0.0	-3.0..+3.1	1.0
Intern-Decision-4B, drop-in	wyrd	cand-single-vs-semif-rotations	71/84	74/84	4	1	+3.6	-1.2..+8.3	0.375
JevK5 v0.2, native	pooled	single	217/259	229/259	25	13	+4.6	-0.8..+9.9	0.073
JevK5 v0.2, native	pooled	rotations	232/259	228/259	9	13	-1.5	-4.9..+1.9	0.5235
JevK5 v0.2, native	authored144	single	116/144	132/144	23	7	+11.1	+2.8..+18.8	0.0052
JevK5 v0.2, native	authored144	rotations	131/144	131/144	7	7	+0.0	-4.9..+4.9	1.0
JevK5 v0.2, native	cicada-w1	single	30/31	30/31	0	0	+0.0	+0.0..+0.0	1.0
JevK5 v0.2, native	cicada-w1	rotations	30/31	30/31	0	0	+0.0	+0.0..+0.0	1.0
JevK5 v0.2, native	wyrd	single	71/84	67/84	2	6	-4.8	-10.7..+1.2	0.2891
JevK5 v0.2, native	wyrd	rotations	71/84	67/84	2	6	-4.8	-10.7..+1.2	0.2891
JevK5 v0.2, native	perturbations108	single	83/108	85/108	10	8	+1.9	-7.4..+11.1	0.8145
JevK5 v0.2, native	perturbations108	rotations	93/108	91/108	4	6	-1.9	-10.2..+5.6	0.7539
JevK5 v0.2, native	cicada-w2	single	20/31	16/31	0	4	-12.9	-25.8..-3.2	0.125
JevK5 v0.2, native	cicada-w2	rotations	19/31	16/31	0	3	-9.7	-19.4..+0.0	0.25
JevK5 v0.2, native	pooled	cand-single-vs-semif-rotations	232/259	229/259	11	14	-1.2	-5.0..+2.6	0.69
JevK5 v0.2, native	wyrd	cand-single-vs-semif-rotations	71/84	67/84	3	7	-4.8	-13.1..+2.4	0.3438
JevK5 v0.2, drop-in	pooled	single	217/259	232/259	24	9	+5.8	+1.1..+10.7	0.0135
JevK5 v0.2, drop-in	pooled	rotations	232/259	237/259	11	6	+1.9	-1.1..+5.1	0.3323
JevK5 v0.2, drop-in	authored144	single	116/144	130/144	21	7	+9.7	+2.1..+17.4	0.0125
JevK5 v0.2, drop-in	authored144	rotations	131/144	133/144	6	4	+1.4	-2.8..+5.6	0.7539
JevK5 v0.2, drop-in	cicada-w1	single	30/31	30/31	0	0	+0.0	+0.0..+0.0	1.0
JevK5 v0.2, drop-in	cicada-w1	rotations	30/31	30/31	0	0	+0.0	+0.0..+0.0	1.0
JevK5 v0.2, drop-in	wyrd	single	71/84	72/84	3	2	+1.2	-3.6..+6.0	1.0
JevK5 v0.2, drop-in	wyrd	rotations	71/84	74/84	5	2	+3.6	-2.4..+10.7	0.4531
JevK5 v0.2, drop-in	perturbations108	single	83/108	92/108	13	4	+8.3	-0.9..+17.6	0.049
JevK5 v0.2, drop-in	perturbations108	rotations	93/108	93/108	4	4	+0.0	-7.4..+6.5	1.0
JevK5 v0.2, drop-in	cicada-w2	single	20/31	17/31	0	3	-9.7	-22.6..+0.0	0.25
JevK5 v0.2, drop-in	cicada-w2	rotations	19/31	16/31	0	3	-9.7	-19.4..+0.0	0.25
JevK5 v0.2, drop-in	pooled	cand-single-vs-semif-rotations	232/259	232/259	10	10	+0.0	-3.2..+3.2	1.0
JevK5 v0.2, drop-in	wyrd	cand-single-vs-semif-rotations	71/84	72/84	3	2	+1.2	-3.6..+6.0	1.0
JevK5 v0.3, native	pooled	single	217/259	227/259	25	15	+3.9	-0.9..+8.5	0.1539
JevK5 v0.3, native	pooled	rotations	232/259	226/259	9	15	-2.3	-6.0..+1.4	0.3075
JevK5 v0.3, native	authored144	single	116/144	135/144	24	5	+13.2	+6.9..+19.4	0.0005
JevK5 v0.3, native	authored144	rotations	131/144	135/144	7	3	+2.8	-1.4..+7.6	0.3438
JevK5 v0.3, native	cicada-w1	single	30/31	26/31	0	4	-12.9	-25.8..-3.2	0.125
JevK5 v0.3, native	cicada-w1	rotations	30/31	26/31	0	4	-12.9	-25.8..-3.2	0.125
JevK5 v0.3, native	wyrd	single	71/84	66/84	1	6	-6.0	-10.7..-1.2	0.125
JevK5 v0.3, native	wyrd	rotations	71/84	65/84	2	8	-7.1	-13.1..-1.2	0.1094
JevK5 v0.3, native	perturbations108	single	83/108	101/108	18	0	+16.7	+8.3..+25.9	0.0
JevK5 v0.3, native	perturbations108	rotations	93/108	101/108	8	0	+7.4	+1.9..+13.9	0.0078
JevK5 v0.3, native	cicada-w2	single	20/31	17/31	0	3	-9.7	-19.4..+0.0	0.25
JevK5 v0.3, native	cicada-w2	rotations	19/31	16/31	0	3	-9.7	-19.4..+0.0	0.25
JevK5 v0.3, native	pooled	cand-single-vs-semif-rotations	232/259	227/259	11	16	-1.9	-5.5..+1.5	0.4421
JevK5 v0.3, native	wyrd	cand-single-vs-semif-rotations	71/84	66/84	2	7	-6.0	-11.9..+0.0	0.1797
JevK5 v0.3, drop-in	pooled	single	217/259	227/259	24	14	+3.9	-0.8..+8.2	0.1433
JevK5 v0.3, drop-in	pooled	rotations	232/259	227/259	10	15	-1.9	-5.5..+1.8	0.4244
JevK5 v0.3, drop-in	authored144	single	116/144	132/144	22	6	+11.1	+4.9..+17.4	0.0037
JevK5 v0.3, drop-in	authored144	rotations	131/144	135/144	8	4	+2.8	-1.4..+7.6	0.3877
JevK5 v0.3, drop-in	cicada-w1	single	30/31	26/31	0	4	-12.9	-25.8..-3.2	0.125
JevK5 v0.3, drop-in	cicada-w1	rotations	30/31	26/31	0	4	-12.9	-25.8..-3.2	0.125
JevK5 v0.3, drop-in	wyrd	single	71/84	69/84	2	4	-2.4	-7.1..+2.4	0.6875
JevK5 v0.3, drop-in	wyrd	rotations	71/84	66/84	2	7	-6.0	-11.9..+0.0	0.1797
JevK5 v0.3, drop-in	perturbations108	single	83/108	99/108	17	1	+14.8	+6.5..+24.1	0.0001
JevK5 v0.3, drop-in	perturbations108	rotations	93/108	101/108	8	0	+7.4	+1.9..+13.9	0.0078
JevK5 v0.3, drop-in	cicada-w2	single	20/31	21/31	1	0	+3.2	+0.0..+9.7	1.0
JevK5 v0.3, drop-in	cicada-w2	rotations	19/31	19/31	0	0	+0.0	+0.0..+0.0	1.0
JevK5 v0.3, drop-in	pooled	cand-single-vs-semif-rotations	232/259	227/259	10	15	-1.9	-5.2..+1.2	0.4244
JevK5 v0.3, drop-in	wyrd	cand-single-vs-semif-rotations	71/84	69/84	3	5	-2.4	-8.3..+3.6	0.7266

Noise floors

system	A-vs-A in process (authored144): flips, max Δp	across restarts, single (560 rows/pair): flips per pair, max Δp	across restarts, rotations: flips per pair	labelled rows whose top moved in ANY restart pair, single: pooled /259 · perturb /108 · Cicada w2 /31
SemIf (Qwen3.5-4B), baseline	0/144 Δp≤0.000, 0/144 Δp≤0.000, 0/144 Δp≤0.000, 0/144 Δp≤0.000	0 (Δp≤0.000), 0 (Δp≤0.000), 0 (Δp≤0.000), 0 (Δp≤0.000), 0 (Δp≤0.000), 0 (Δp≤0.000)	0, 0, 0, 0, 0, 0	0 · 0 · 0 (4 repeats)
Plumb-4B, native	0/144 Δp≤0.000, 0/144 Δp≤0.000, 0/144 Δp≤0.000	0 (Δp≤0.000), 2 (Δp≤0.043), 2 (Δp≤0.043)	0, 2, 2	2 · 0 · 0 (3 repeats)
Plumb-4B, drop-in	0/144 Δp≤0.000, 0/144 Δp≤0.000, 0/144 Δp≤0.000	0 (Δp≤0.000), 0 (Δp≤0.000), 0 (Δp≤0.000)	0, 0, 0	0 · 0 · 0 (3 repeats)
Imajev-4B, native	0/144 Δp≤0.000, 0/144 Δp≤0.000, 0/144 Δp≤0.000, 0/144 Δp≤0.000	0 (Δp≤0.000), 0 (Δp≤0.000), 0 (Δp≤0.000), 0 (Δp≤0.000), 0 (Δp≤0.000), 0 (Δp≤0.000)	0, 0, 0, 0, 0, 0	0 · 0 · 0 (4 repeats)
Intern-Decision-4B, native	0/144 Δp≤0.000, 0/144 Δp≤0.000, 0/144 Δp≤0.000, 0/144 Δp≤0.000	0 (Δp≤0.000), 0 (Δp≤0.000), 0 (Δp≤0.000), 0 (Δp≤0.000), 0 (Δp≤0.000), 0 (Δp≤0.000)	0, 0, 0, 0, 0, 0	0 · 0 · 0 (4 repeats)
Intern-Decision-4B, drop-in	0/144 Δp≤0.000, 0/144 Δp≤0.000, 0/144 Δp≤0.000	0 (Δp≤0.128), 0 (Δp≤0.128), 0 (Δp≤0.000)	0, 0, 0	0 · 0 · 0 (3 repeats)
JevK5 v0.2, native	0/144 Δp≤0.000, 0/144 Δp≤0.000, 0/144 Δp≤0.000	0 (Δp≤0.000), 3 (Δp≤0.036), 3 (Δp≤0.036)	0, 2, 2	1 · 1 · 0 (3 repeats)
JevK5 v0.2, drop-in	0/144 Δp≤0.000, 0/144 Δp≤0.000, 0/144 Δp≤0.000	0 (Δp≤0.000), 0 (Δp≤0.000), 0 (Δp≤0.000)	0, 0, 0	0 · 0 · 0 (3 repeats)
JevK5 v0.3, native	0/144 Δp≤0.000, 0/144 Δp≤0.000, 0/144 Δp≤0.000	1 (Δp≤0.051), 1 (Δp≤0.051), 0 (Δp≤0.000)	1, 1, 0	0 · 1 · 0 (3 repeats)
JevK5 v0.3, drop-in	0/144 Δp≤0.000, 0/144 Δp≤0.000, 0/144 Δp≤0.000	0 (Δp≤0.000), 0 (Δp≤0.000), 0 (Δp≤0.000)	0, 0, 0	0 · 0 · 0 (3 repeats)

Order sensitivity (authored144, single ordering vs the same request reordered) and the negative control

system	reversed: label changes /144	reversed: max Δp	shuffled: label changes	shuffled: max Δp	NEG: same top as unrotated /144	NEG: follows the description	NEG: right vs original gold
SemIf (Qwen3.5-4B), baseline	30	0.89	29	0.89	14	119	17
Plumb-4B, native	15	0.54 (0.54–0.56)	10	0.44 (0.44–0.45)	24 (24–25)	106 (105–106)	27 (27–28)
Plumb-4B, drop-in	12	0.73	8	0.67	3	126	12
Imajev-4B, native	3	0.72	4	0.68	4	135	5
Intern-Decision-4B, native	8	0.60	5	0.46	10	122	14
Intern-Decision-4B, drop-in	10	0.98	12 (11–12)	0.94 (0.93–0.94)	5	127	10
JevK5 v0.2, native	13 (12–13)	0.72	10	0.64 (0.63–0.64)	20	114	21
JevK5 v0.2, drop-in	15	0.70	12	0.70	4	128	9
JevK5 v0.3, native	8 (7–8)	0.61	11	0.49	20	118	18
JevK5 v0.3, drop-in	4	0.53	5	0.51	4	133	5

Latency at our shape (loopback on fv-ml1, GPU 3; ms; median of all requests, run-median range)

system	short1 e2e	crit21 e2e	crit21 server	long1 e2e	long16 e2e	long16 server	tokens crit21 / long16
SemIf (Qwen3.5-4B), baseline	37 (37–38)	131 (131–132)	119 (119–120)	186 (186–187)	505 (502–519)	486 (482–500)	62 / 3900
Plumb-4B, native	16	294 (292–295)	292 (290–293)	215 (212–215)	3378 (3374–3384)	3376 (3372–3382)	2489 / 63302
Plumb-4B, drop-in	37	131 (131–132)	120 (119–120)	186	506 (506–507)	487 (487–488)	62 / 3900
Imajev-4B, native	60 (60–61)	1219 (1212–1230)	1212 (1204–1223)	315 (310–315)	5125 (5054–5140)	5121 (5049–5135)	374 / 7925
Intern-Decision-4B, native	39	88 (88–89)	84 (84–85)	190 (189–191)	215 (213–216)	213 (211–214)	1083 / 4579
Intern-Decision-4B, drop-in	38 (37–38)	132	120	186 (186–188)	508 (500–516)	488 (481–497)	62 / 3900
JevK5 v0.2, native	16	291 (291–292)	289 (289–290)	212	3360 (3358–3368)	3358 (3356–3366)	2489 / 63302
JevK5 v0.2, drop-in	38	132	120 (119–120)	187 (185–188)	518 (505–519)	499 (486–500)	62 / 3900
JevK5 v0.3, native	16	291 (291–292)	289 (289–290)	212	3363 (3355–3370)	3361 (3353–3368)	2489 / 63302
JevK5 v0.3, drop-in	38 (37–39)	133 (131–133)	120 (119–121)	187 (186–188)	518 (506–521)	498 (487–501)	62 / 3900

VRAM (nvidia-smi, whole GPU 3, only our process on it) and capacity under the 12 GiB cap

system	rest after warm-up MiB	peak in sets+shapes MiB	peak in capacity sweep MiB	max rows @ short state	max rows @ ~3,900 tok
SemIf (Qwen3.5-4B), baseline	9242 (9242–9250)	12918 (12886–12918)	12920	64 (next 96: 422)	16 (next 18: 503)
Plumb-4B, native	12064	12064	12064	128 (no failure up to the last step)	64 (no failure up to the last step)
Plumb-4B, drop-in	9242	12918	12920	64 (next 96: 422)	16 (next 18: 503)
Imajev-4B, native	10404	11056	11056	128 (no failure up to the last step)	64 (no failure up to the last step)
Intern-Decision-4B, native	9736	10290	10290	128 (no failure up to the last step)	64 (no failure up to the last step)
Intern-Decision-4B, drop-in	9242	12918	12920	64 (next 96: 422)	16 (next 18: 503)
JevK5 v0.2, native	12064	12064	12064	128 (no failure up to the last step)	64 (no failure up to the last step)
JevK5 v0.2, drop-in	9242	12918	–	–	–
JevK5 v0.3, native	12064	12064	–	–	–
JevK5 v0.3, drop-in	9242	12918	–	–	–

25 KiB Raw Blame History Unescape Escape

Key table

JevBench v1.2.16 public items (positive control + candidate score)

Replaced-baseline sets, single (caller order)

Replaced-baseline sets, rotations (n rotations averaged)

Wyrd as one request per turn (4 decisions over one state: SemIf /decide/shared, native multi-question request)

Null control (content-free state) and positive controls inside the spike sets, single ordering

Paired against SemIf (each row's majority top over the repeats; group bootstrap 95% CI; exact McNemar)

Noise floors

Order sensitivity (authored144, single ordering vs the same request reordered) and the negative control

Latency at our shape (loopback on fv-ml1, GPU 3; ms; median of all requests, run-median range)

VRAM (nvidia-smi, whole GPU 3, only our process on it) and capacity under the 12 GiB cap

25 KiB

Raw Blame History