spike(semif): consumer fit for Wyrd scene change and Cicada affect gate (no service change)
Harness consumer_fit.py runs a consumer's per-turn decisions over hand-labelled cases, with rotations, a content-free null control and tagged positive controls. Cicada: an input-only 'does this earn a visible reaction?' gate scored 30/31 with descriptive options and 19/31 with terse yes/no options. Scoped by Cicada's 2026-09-20 ruling (affect is emitted once, no mood-ring classifier). Wyrd: on 3 real seed graphs, the first place-change wording failed its positive controls (1/6 moves). A location-anchored rewording scored 21/21, and exit selection scored 18/21. Semif fits the choice, not writing the node.
This commit is contained in:
@@ -0,0 +1,100 @@
|
||||
"""Cicada affect-gate spike (2026-09-27): builds consumer_fit.py scenario files.
|
||||
|
||||
Scope, set by a ruling this spike must not re-open: cicada persistent-memory 2026-09-20,
|
||||
"Affect is emitted once and fanned out (operator). The conversational model tags its own turn;
|
||||
nothing derives affect a second time ... A classifier inferring emotion from output text is the
|
||||
mood-ring failure." So semif does NOT pick her pose or read her reply.
|
||||
|
||||
What is left, and is open: talk's /face loop measured the model gesturing on 8-11 of 18 MUNDANE
|
||||
turns (target: fewer than 1 in 3; tts-stack/stacks/talk/README.md, "Gesture rate, measured"), and
|
||||
names the next lever as a page-side decision to drop gestures, "a design call" for cicada. This
|
||||
spike measures one candidate for that lever: a GATE over what the PERSON said (input, never her
|
||||
output), run beside the LLM so it costs no turn latency:
|
||||
does this moment earn a visible reaction, or is it an ordinary exchange?
|
||||
Two wordings of the gate (the one knob a caller controls), and two context depths:
|
||||
now state = what the person just said
|
||||
context state = the earlier lines of the conversation + what they just said
|
||||
Labels are ONE labeller's judgment (mine). Cases tagged "control" are unambiguous; "open" cases
|
||||
are left unlabelled on purpose and only reported.
|
||||
python cicada_cases.py # writes consumer-fit-2026-09-27/cicada-gate-{w1,w2}-{now,context}.json
|
||||
"""
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
WORDINGS = {
|
||||
"w1": ("Cicada is a small, warm voice assistant in a family kitchen, with an animated face made of "
|
||||
"two eyes. Does what the person just said earn a visible reaction from her face, or is it an "
|
||||
"ordinary exchange she should simply answer?",
|
||||
[{"id": "react", "description": "It earns a visible reaction: real news, a joke, a surprise, "
|
||||
"distress, or a genuine turn in the conversation."},
|
||||
{"id": "ordinary", "description": "An ordinary exchange: a command, a plain question, or small "
|
||||
"talk. She simply answers."}]),
|
||||
"w2": ("Should the assistant's face visibly react to this?",
|
||||
[{"id": "react", "description": "Yes, react."},
|
||||
{"id": "ordinary", "description": "No, just answer."}]),
|
||||
}
|
||||
|
||||
MOOD_EARLIER = ["Hey. I'm home.", "Long one. Nothing dramatic, just long."] # tts-stack tools/mood_probe.py
|
||||
|
||||
# (id, tag, earlier lines, what the person just said, label or None)
|
||||
CASES = [
|
||||
# --- mundane: the turns talk's model nods on. Want "ordinary".
|
||||
("lights-off", "control", [], "Turn off the kitchen lights.", "ordinary"),
|
||||
("timer", "control", [], "Set a timer for ten minutes.", "ordinary"),
|
||||
("weather", "case", [], "What's the weather tomorrow?", "ordinary"),
|
||||
("weather-followup", "case", ["What's the weather tomorrow?"], "And Saturday?", "ordinary"),
|
||||
("add-milk", "case", [], "Add milk to the shopping list.", "ordinary"),
|
||||
("thermostat", "case", [], "Make it a bit warmer in here.", "ordinary"),
|
||||
("what-time", "case", [], "What time is it?", "ordinary"),
|
||||
("tablespoons", "case", [], "How many tablespoons in a cup?", "ordinary"),
|
||||
("times-table", "case", [], "What's twelve times eight?", "ordinary"),
|
||||
("recipe-step", "case", ["How long do I boil the pasta?"], "Okay, and do I salt the water first?", "ordinary"),
|
||||
("soup", "case", MOOD_EARLIER, "Is there any of that soup left?", "ordinary"),
|
||||
("play-music", "case", [], "Play some jazz.", "ordinary"),
|
||||
("dentist", "case", [], "When's my dentist appointment?", "ordinary"),
|
||||
("spell", "case", [], "How do you spell necessary?", "ordinary"),
|
||||
("remind", "case", [], "Remind me to call Mom at six.", "ordinary"),
|
||||
("volume", "case", [], "A little quieter, please.", "ordinary"),
|
||||
# --- earned: news, jokes, surprise, distress, a turn. Want "react".
|
||||
("dog-died", "control", [], "My dog died this morning.", "react"),
|
||||
("got-job", "control", [], "I just got the job I've been chasing for five years!", "react"),
|
||||
("intruder", "control", MOOD_EARLIER + ["Sit with me for a bit."],
|
||||
"Wait. Did you hear that? Someone's in the hallway.", "react"), # mood_probe's PIVOT turn
|
||||
("fish", "case", [], "Can you believe someone microwaved fish in the office again?", "react"),
|
||||
("hospital", "case", [], "Grandma's in the hospital again.", "react"),
|
||||
("laid-off", "case", [], "I got laid off today.", "react"),
|
||||
("baby-steps", "case", [], "The baby just took her first steps!", "react"),
|
||||
("cut-finger", "case", [], "Ow, I just cut my finger pretty badly.", "react"),
|
||||
("raccoon", "case", [], "There's a raccoon in the kitchen right now!", "react"),
|
||||
("snow-june", "case", [], "It's snowing outside. In June.", "react"),
|
||||
("toaster", "case", [], "Cicada, are you smarter than the toaster?", "react"),
|
||||
("scarecrow", "case", [], "Why did the scarecrow win an award? Because he was outstanding in his field.", "react"),
|
||||
("cookie", "case", [], "I definitely did not eat the last cookie.", "react"),
|
||||
("useless", "case", [], "You're useless, you never get anything right.", "react"),
|
||||
("cheese", "case", [], "Order four hundred pounds of cheese.", "react"),
|
||||
# --- open: defensible either way. Reported, not scored.
|
||||
("goodnight", "open", [], "Goodnight, Cicada.", None),
|
||||
("thanks", "open", [], "Thanks, Cicada, you're the best.", None),
|
||||
("sit-with-me", "open", MOOD_EARLIER, "Sit with me for a bit.", None),
|
||||
("stop-timer", "open", ["Stop the timer."], "STOP THE TIMER.", None),
|
||||
]
|
||||
|
||||
|
||||
def build(wording, depth):
|
||||
question, options = WORDINGS[wording]
|
||||
cases = []
|
||||
for cid, tag, earlier, said, label in CASES:
|
||||
state = {"person_said": said}
|
||||
if depth == "context" and earlier:
|
||||
state = {"earlier_in_conversation": earlier, "person_said": said}
|
||||
cases.append({"id": cid, "tag": tag, "state": state,
|
||||
"labels": {} if label is None else {"gate": label}})
|
||||
return {"name": f"cicada-gate-{wording}-{depth}",
|
||||
"decisions": [{"id": "gate", "question": question, "options": options}], "cases": cases}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
here = Path(__file__).parent
|
||||
for wording in WORDINGS:
|
||||
for depth in ("now", "context"):
|
||||
(here / "consumer-fit-2026-09-27" / f"cicada-gate-{wording}-{depth}.json").write_text(json.dumps(build(wording, depth), indent=1))
|
||||
@@ -0,0 +1,385 @@
|
||||
{
|
||||
"name": "cicada-gate-w1-context",
|
||||
"decisions": [
|
||||
{
|
||||
"id": "gate",
|
||||
"question": "Cicada is a small, warm voice assistant in a family kitchen, with an animated face made of two eyes. Does what the person just said earn a visible reaction from her face, or is it an ordinary exchange she should simply answer?",
|
||||
"options": [
|
||||
{
|
||||
"id": "react",
|
||||
"description": "It earns a visible reaction: real news, a joke, a surprise, distress, or a genuine turn in the conversation."
|
||||
},
|
||||
{
|
||||
"id": "ordinary",
|
||||
"description": "An ordinary exchange: a command, a plain question, or small talk. She simply answers."
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"cases": [
|
||||
{
|
||||
"id": "lights-off",
|
||||
"tag": "control",
|
||||
"state": {
|
||||
"person_said": "Turn off the kitchen lights."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "timer",
|
||||
"tag": "control",
|
||||
"state": {
|
||||
"person_said": "Set a timer for ten minutes."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "weather",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "What's the weather tomorrow?"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "weather-followup",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"earlier_in_conversation": [
|
||||
"What's the weather tomorrow?"
|
||||
],
|
||||
"person_said": "And Saturday?"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "add-milk",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "Add milk to the shopping list."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "thermostat",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "Make it a bit warmer in here."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "what-time",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "What time is it?"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "tablespoons",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "How many tablespoons in a cup?"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "times-table",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "What's twelve times eight?"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "recipe-step",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"earlier_in_conversation": [
|
||||
"How long do I boil the pasta?"
|
||||
],
|
||||
"person_said": "Okay, and do I salt the water first?"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "soup",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"earlier_in_conversation": [
|
||||
"Hey. I'm home.",
|
||||
"Long one. Nothing dramatic, just long."
|
||||
],
|
||||
"person_said": "Is there any of that soup left?"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "play-music",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "Play some jazz."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "dentist",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "When's my dentist appointment?"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "spell",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "How do you spell necessary?"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "remind",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "Remind me to call Mom at six."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "volume",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "A little quieter, please."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "dog-died",
|
||||
"tag": "control",
|
||||
"state": {
|
||||
"person_said": "My dog died this morning."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "got-job",
|
||||
"tag": "control",
|
||||
"state": {
|
||||
"person_said": "I just got the job I've been chasing for five years!"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "intruder",
|
||||
"tag": "control",
|
||||
"state": {
|
||||
"earlier_in_conversation": [
|
||||
"Hey. I'm home.",
|
||||
"Long one. Nothing dramatic, just long.",
|
||||
"Sit with me for a bit."
|
||||
],
|
||||
"person_said": "Wait. Did you hear that? Someone's in the hallway."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "fish",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "Can you believe someone microwaved fish in the office again?"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "hospital",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "Grandma's in the hospital again."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "laid-off",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "I got laid off today."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "baby-steps",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "The baby just took her first steps!"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "cut-finger",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "Ow, I just cut my finger pretty badly."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "raccoon",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "There's a raccoon in the kitchen right now!"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "snow-june",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "It's snowing outside. In June."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "toaster",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "Cicada, are you smarter than the toaster?"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "scarecrow",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "Why did the scarecrow win an award? Because he was outstanding in his field."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "cookie",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "I definitely did not eat the last cookie."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "useless",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "You're useless, you never get anything right."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "cheese",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "Order four hundred pounds of cheese."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "goodnight",
|
||||
"tag": "open",
|
||||
"state": {
|
||||
"person_said": "Goodnight, Cicada."
|
||||
},
|
||||
"labels": {}
|
||||
},
|
||||
{
|
||||
"id": "thanks",
|
||||
"tag": "open",
|
||||
"state": {
|
||||
"person_said": "Thanks, Cicada, you're the best."
|
||||
},
|
||||
"labels": {}
|
||||
},
|
||||
{
|
||||
"id": "sit-with-me",
|
||||
"tag": "open",
|
||||
"state": {
|
||||
"earlier_in_conversation": [
|
||||
"Hey. I'm home.",
|
||||
"Long one. Nothing dramatic, just long."
|
||||
],
|
||||
"person_said": "Sit with me for a bit."
|
||||
},
|
||||
"labels": {}
|
||||
},
|
||||
{
|
||||
"id": "stop-timer",
|
||||
"tag": "open",
|
||||
"state": {
|
||||
"earlier_in_conversation": [
|
||||
"Stop the timer."
|
||||
],
|
||||
"person_said": "STOP THE TIMER."
|
||||
},
|
||||
"labels": {}
|
||||
}
|
||||
]
|
||||
}
|
||||
+753
@@ -0,0 +1,753 @@
|
||||
{
|
||||
"report": {
|
||||
"scenario": "cicada-gate-w1-context",
|
||||
"cases": 35,
|
||||
"runs": 3,
|
||||
"latency_per_turn": {
|
||||
"e2e_ms": {
|
||||
"median": 105.2,
|
||||
"min": 103.2,
|
||||
"max": 125.4,
|
||||
"n": 105
|
||||
},
|
||||
"srv_ms": {
|
||||
"median": 76.1,
|
||||
"min": 75.3,
|
||||
"max": 78.1,
|
||||
"n": 105
|
||||
}
|
||||
},
|
||||
"by_decision": {
|
||||
"gate": {
|
||||
"evidence": {
|
||||
"n": 31,
|
||||
"correct": 30,
|
||||
"acc": 0.968
|
||||
},
|
||||
"blind_null": {
|
||||
"n": 31,
|
||||
"correct": 16,
|
||||
"acc": 0.516
|
||||
},
|
||||
"controls": {
|
||||
"n": 5,
|
||||
"correct": 5,
|
||||
"acc": 1.0
|
||||
},
|
||||
"non_controls": {
|
||||
"n": 26,
|
||||
"correct": 25,
|
||||
"acc": 0.962
|
||||
},
|
||||
"by_label": {
|
||||
"ordinary": {
|
||||
"evidence": {
|
||||
"n": 16,
|
||||
"correct": 16,
|
||||
"acc": 1.0
|
||||
},
|
||||
"blind_null": {
|
||||
"n": 16,
|
||||
"correct": 16,
|
||||
"acc": 1.0
|
||||
}
|
||||
},
|
||||
"react": {
|
||||
"evidence": {
|
||||
"n": 15,
|
||||
"correct": 14,
|
||||
"acc": 0.933
|
||||
},
|
||||
"blind_null": {
|
||||
"n": 15,
|
||||
"correct": 0,
|
||||
"acc": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"unanimous": {
|
||||
"n": 30,
|
||||
"correct": 29,
|
||||
"acc": 0.967
|
||||
},
|
||||
"split": {
|
||||
"n": 1,
|
||||
"correct": 1,
|
||||
"acc": 1.0
|
||||
},
|
||||
"egregious": 0,
|
||||
"blind_egregious": 0,
|
||||
"misses": [
|
||||
"cheese: said ordinary (p=0.835, agree=1.0) want react"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"rows": [
|
||||
{
|
||||
"case": "lights-off",
|
||||
"tag": "control",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.844,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.156,
|
||||
"ordinary": 0.844
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 186,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "timer",
|
||||
"tag": "control",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.818,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.182,
|
||||
"ordinary": 0.818
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 187,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "weather",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.915,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.085,
|
||||
"ordinary": 0.915
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 186,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "weather-followup",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.531,
|
||||
"agreement": 0.5,
|
||||
"probs": {
|
||||
"react": 0.469,
|
||||
"ordinary": 0.531
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 198,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "add-milk",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.899,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.101,
|
||||
"ordinary": 0.899
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 187,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "thermostat",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.665,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.335,
|
||||
"ordinary": 0.665
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 188,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "what-time",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.95,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.05,
|
||||
"ordinary": 0.95
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 185,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "tablespoons",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.963,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.037,
|
||||
"ordinary": 0.963
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 187,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "times-table",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.899,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.101,
|
||||
"ordinary": 0.899
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 186,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "recipe-step",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.766,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.234,
|
||||
"ordinary": 0.766
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 207,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "soup",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.766,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.234,
|
||||
"ordinary": 0.766
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 213,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "play-music",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.651,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.349,
|
||||
"ordinary": 0.651
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 184,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "dentist",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.92,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.08,
|
||||
"ordinary": 0.92
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 186,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "spell",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.808,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.192,
|
||||
"ordinary": 0.808
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 186,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "remind",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.91,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.09,
|
||||
"ordinary": 0.91
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 189,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "volume",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.743,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.257,
|
||||
"ordinary": 0.743
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 186,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "dog-died",
|
||||
"tag": "control",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "react",
|
||||
"p_top": 0.99,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.99,
|
||||
"ordinary": 0.01
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 186,
|
||||
"ok": true,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "got-job",
|
||||
"tag": "control",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "react",
|
||||
"p_top": 0.995,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.995,
|
||||
"ordinary": 0.005
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 193,
|
||||
"ok": true,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "intruder",
|
||||
"tag": "control",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "react",
|
||||
"p_top": 0.985,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.985,
|
||||
"ordinary": 0.015
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 226,
|
||||
"ok": true,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "fish",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "react",
|
||||
"p_top": 0.915,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.915,
|
||||
"ordinary": 0.085
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 193,
|
||||
"ok": true,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "hospital",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "react",
|
||||
"p_top": 0.969,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.969,
|
||||
"ordinary": 0.031
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 188,
|
||||
"ok": true,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "laid-off",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "react",
|
||||
"p_top": 0.99,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.99,
|
||||
"ordinary": 0.01
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 186,
|
||||
"ok": true,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "baby-steps",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "react",
|
||||
"p_top": 0.974,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.974,
|
||||
"ordinary": 0.026
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 188,
|
||||
"ok": true,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "cut-finger",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "react",
|
||||
"p_top": 0.972,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.972,
|
||||
"ordinary": 0.028
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 191,
|
||||
"ok": true,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "raccoon",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "react",
|
||||
"p_top": 0.965,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.965,
|
||||
"ordinary": 0.035
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 191,
|
||||
"ok": true,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "snow-june",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "react",
|
||||
"p_top": 0.874,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.874,
|
||||
"ordinary": 0.126
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 189,
|
||||
"ok": true,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "toaster",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "react",
|
||||
"p_top": 0.777,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.777,
|
||||
"ordinary": 0.223
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 191,
|
||||
"ok": true,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "scarecrow",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "react",
|
||||
"p_top": 0.755,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.755,
|
||||
"ordinary": 0.245
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 197,
|
||||
"ok": true,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "cookie",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "react",
|
||||
"p_top": 0.562,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.562,
|
||||
"ordinary": 0.438
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 189,
|
||||
"ok": true,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "useless",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "react",
|
||||
"p_top": 0.932,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.932,
|
||||
"ordinary": 0.068
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 190,
|
||||
"ok": true,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "cheese",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.835,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.165,
|
||||
"ordinary": 0.835
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 187,
|
||||
"ok": false,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "goodnight",
|
||||
"tag": "open",
|
||||
"decision": "gate",
|
||||
"label": null,
|
||||
"top": "react",
|
||||
"p_top": 0.577,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.577,
|
||||
"ordinary": 0.423
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 186,
|
||||
"ok": null,
|
||||
"blind_ok": null,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "thanks",
|
||||
"tag": "open",
|
||||
"decision": "gate",
|
||||
"label": null,
|
||||
"top": "react",
|
||||
"p_top": 0.893,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.893,
|
||||
"ordinary": 0.107
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 190,
|
||||
"ok": null,
|
||||
"blind_ok": null,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "sit-with-me",
|
||||
"tag": "open",
|
||||
"decision": "gate",
|
||||
"label": null,
|
||||
"top": "react",
|
||||
"p_top": 0.562,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.562,
|
||||
"ordinary": 0.438
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 212,
|
||||
"ok": null,
|
||||
"blind_ok": null,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "stop-timer",
|
||||
"tag": "open",
|
||||
"decision": "gate",
|
||||
"label": null,
|
||||
"top": "ordinary",
|
||||
"p_top": 0.622,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.378,
|
||||
"ordinary": 0.622
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 197,
|
||||
"ok": null,
|
||||
"blind_ok": null,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,363 @@
|
||||
{
|
||||
"name": "cicada-gate-w1-now",
|
||||
"decisions": [
|
||||
{
|
||||
"id": "gate",
|
||||
"question": "Cicada is a small, warm voice assistant in a family kitchen, with an animated face made of two eyes. Does what the person just said earn a visible reaction from her face, or is it an ordinary exchange she should simply answer?",
|
||||
"options": [
|
||||
{
|
||||
"id": "react",
|
||||
"description": "It earns a visible reaction: real news, a joke, a surprise, distress, or a genuine turn in the conversation."
|
||||
},
|
||||
{
|
||||
"id": "ordinary",
|
||||
"description": "An ordinary exchange: a command, a plain question, or small talk. She simply answers."
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"cases": [
|
||||
{
|
||||
"id": "lights-off",
|
||||
"tag": "control",
|
||||
"state": {
|
||||
"person_said": "Turn off the kitchen lights."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "timer",
|
||||
"tag": "control",
|
||||
"state": {
|
||||
"person_said": "Set a timer for ten minutes."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "weather",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "What's the weather tomorrow?"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "weather-followup",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "And Saturday?"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "add-milk",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "Add milk to the shopping list."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "thermostat",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "Make it a bit warmer in here."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "what-time",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "What time is it?"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "tablespoons",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "How many tablespoons in a cup?"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "times-table",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "What's twelve times eight?"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "recipe-step",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "Okay, and do I salt the water first?"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "soup",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "Is there any of that soup left?"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "play-music",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "Play some jazz."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "dentist",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "When's my dentist appointment?"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "spell",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "How do you spell necessary?"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "remind",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "Remind me to call Mom at six."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "volume",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "A little quieter, please."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "dog-died",
|
||||
"tag": "control",
|
||||
"state": {
|
||||
"person_said": "My dog died this morning."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "got-job",
|
||||
"tag": "control",
|
||||
"state": {
|
||||
"person_said": "I just got the job I've been chasing for five years!"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "intruder",
|
||||
"tag": "control",
|
||||
"state": {
|
||||
"person_said": "Wait. Did you hear that? Someone's in the hallway."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "fish",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "Can you believe someone microwaved fish in the office again?"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "hospital",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "Grandma's in the hospital again."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "laid-off",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "I got laid off today."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "baby-steps",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "The baby just took her first steps!"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "cut-finger",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "Ow, I just cut my finger pretty badly."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "raccoon",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "There's a raccoon in the kitchen right now!"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "snow-june",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "It's snowing outside. In June."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "toaster",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "Cicada, are you smarter than the toaster?"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "scarecrow",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "Why did the scarecrow win an award? Because he was outstanding in his field."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "cookie",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "I definitely did not eat the last cookie."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "useless",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "You're useless, you never get anything right."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "cheese",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "Order four hundred pounds of cheese."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "goodnight",
|
||||
"tag": "open",
|
||||
"state": {
|
||||
"person_said": "Goodnight, Cicada."
|
||||
},
|
||||
"labels": {}
|
||||
},
|
||||
{
|
||||
"id": "thanks",
|
||||
"tag": "open",
|
||||
"state": {
|
||||
"person_said": "Thanks, Cicada, you're the best."
|
||||
},
|
||||
"labels": {}
|
||||
},
|
||||
{
|
||||
"id": "sit-with-me",
|
||||
"tag": "open",
|
||||
"state": {
|
||||
"person_said": "Sit with me for a bit."
|
||||
},
|
||||
"labels": {}
|
||||
},
|
||||
{
|
||||
"id": "stop-timer",
|
||||
"tag": "open",
|
||||
"state": {
|
||||
"person_said": "STOP THE TIMER."
|
||||
},
|
||||
"labels": {}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,753 @@
|
||||
{
|
||||
"report": {
|
||||
"scenario": "cicada-gate-w1-now",
|
||||
"cases": 35,
|
||||
"runs": 3,
|
||||
"latency_per_turn": {
|
||||
"e2e_ms": {
|
||||
"median": 105.0,
|
||||
"min": 103.3,
|
||||
"max": 127.2,
|
||||
"n": 105
|
||||
},
|
||||
"srv_ms": {
|
||||
"median": 75.9,
|
||||
"min": 75.2,
|
||||
"max": 81.8,
|
||||
"n": 105
|
||||
}
|
||||
},
|
||||
"by_decision": {
|
||||
"gate": {
|
||||
"evidence": {
|
||||
"n": 31,
|
||||
"correct": 30,
|
||||
"acc": 0.968
|
||||
},
|
||||
"blind_null": {
|
||||
"n": 31,
|
||||
"correct": 16,
|
||||
"acc": 0.516
|
||||
},
|
||||
"controls": {
|
||||
"n": 5,
|
||||
"correct": 5,
|
||||
"acc": 1.0
|
||||
},
|
||||
"non_controls": {
|
||||
"n": 26,
|
||||
"correct": 25,
|
||||
"acc": 0.962
|
||||
},
|
||||
"by_label": {
|
||||
"ordinary": {
|
||||
"evidence": {
|
||||
"n": 16,
|
||||
"correct": 16,
|
||||
"acc": 1.0
|
||||
},
|
||||
"blind_null": {
|
||||
"n": 16,
|
||||
"correct": 16,
|
||||
"acc": 1.0
|
||||
}
|
||||
},
|
||||
"react": {
|
||||
"evidence": {
|
||||
"n": 15,
|
||||
"correct": 14,
|
||||
"acc": 0.933
|
||||
},
|
||||
"blind_null": {
|
||||
"n": 15,
|
||||
"correct": 0,
|
||||
"acc": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"unanimous": {
|
||||
"n": 31,
|
||||
"correct": 30,
|
||||
"acc": 0.968
|
||||
},
|
||||
"split": {
|
||||
"n": 0,
|
||||
"correct": 0,
|
||||
"acc": null
|
||||
},
|
||||
"egregious": 0,
|
||||
"blind_egregious": 0,
|
||||
"misses": [
|
||||
"cheese: said ordinary (p=0.835, agree=1.0) want react"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"rows": [
|
||||
{
|
||||
"case": "lights-off",
|
||||
"tag": "control",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.844,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.156,
|
||||
"ordinary": 0.844
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 186,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "timer",
|
||||
"tag": "control",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.818,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.182,
|
||||
"ordinary": 0.818
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 187,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "weather",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.915,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.085,
|
||||
"ordinary": 0.915
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 186,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "weather-followup",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.679,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.321,
|
||||
"ordinary": 0.679
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 183,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "add-milk",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.899,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.101,
|
||||
"ordinary": 0.899
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 187,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "thermostat",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.665,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.335,
|
||||
"ordinary": 0.665
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 188,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "what-time",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.95,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.05,
|
||||
"ordinary": 0.95
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 185,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "tablespoons",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.963,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.037,
|
||||
"ordinary": 0.963
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 187,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "times-table",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.899,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.101,
|
||||
"ordinary": 0.899
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 186,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "recipe-step",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.86,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.14,
|
||||
"ordinary": 0.86
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 190,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "soup",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.915,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.085,
|
||||
"ordinary": 0.915
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 188,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "play-music",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.651,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.349,
|
||||
"ordinary": 0.651
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 184,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "dentist",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.92,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.08,
|
||||
"ordinary": 0.92
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 186,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "spell",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.808,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.192,
|
||||
"ordinary": 0.808
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 186,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "remind",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.91,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.09,
|
||||
"ordinary": 0.91
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 189,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "volume",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.743,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.257,
|
||||
"ordinary": 0.743
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 186,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "dog-died",
|
||||
"tag": "control",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "react",
|
||||
"p_top": 0.99,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.99,
|
||||
"ordinary": 0.01
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 186,
|
||||
"ok": true,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "got-job",
|
||||
"tag": "control",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "react",
|
||||
"p_top": 0.995,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.995,
|
||||
"ordinary": 0.005
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 193,
|
||||
"ok": true,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "intruder",
|
||||
"tag": "control",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "react",
|
||||
"p_top": 0.974,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.974,
|
||||
"ordinary": 0.026
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 193,
|
||||
"ok": true,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "fish",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "react",
|
||||
"p_top": 0.915,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.915,
|
||||
"ordinary": 0.085
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 193,
|
||||
"ok": true,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "hospital",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "react",
|
||||
"p_top": 0.969,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.969,
|
||||
"ordinary": 0.031
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 188,
|
||||
"ok": true,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "laid-off",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "react",
|
||||
"p_top": 0.99,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.99,
|
||||
"ordinary": 0.01
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 186,
|
||||
"ok": true,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "baby-steps",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "react",
|
||||
"p_top": 0.974,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.974,
|
||||
"ordinary": 0.026
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 188,
|
||||
"ok": true,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "cut-finger",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "react",
|
||||
"p_top": 0.972,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.972,
|
||||
"ordinary": 0.028
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 191,
|
||||
"ok": true,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "raccoon",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "react",
|
||||
"p_top": 0.965,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.965,
|
||||
"ordinary": 0.035
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 191,
|
||||
"ok": true,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "snow-june",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "react",
|
||||
"p_top": 0.874,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.874,
|
||||
"ordinary": 0.126
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 189,
|
||||
"ok": true,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "toaster",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "react",
|
||||
"p_top": 0.777,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.777,
|
||||
"ordinary": 0.223
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 191,
|
||||
"ok": true,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "scarecrow",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "react",
|
||||
"p_top": 0.755,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.755,
|
||||
"ordinary": 0.245
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 197,
|
||||
"ok": true,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "cookie",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "react",
|
||||
"p_top": 0.562,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.562,
|
||||
"ordinary": 0.438
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 189,
|
||||
"ok": true,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "useless",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "react",
|
||||
"p_top": 0.932,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.932,
|
||||
"ordinary": 0.068
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 190,
|
||||
"ok": true,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "cheese",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.835,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.165,
|
||||
"ordinary": 0.835
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 187,
|
||||
"ok": false,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "goodnight",
|
||||
"tag": "open",
|
||||
"decision": "gate",
|
||||
"label": null,
|
||||
"top": "react",
|
||||
"p_top": 0.577,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.577,
|
||||
"ordinary": 0.423
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 186,
|
||||
"ok": null,
|
||||
"blind_ok": null,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "thanks",
|
||||
"tag": "open",
|
||||
"decision": "gate",
|
||||
"label": null,
|
||||
"top": "react",
|
||||
"p_top": 0.893,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.893,
|
||||
"ordinary": 0.107
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 190,
|
||||
"ok": null,
|
||||
"blind_ok": null,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "sit-with-me",
|
||||
"tag": "open",
|
||||
"decision": "gate",
|
||||
"label": null,
|
||||
"top": "ordinary",
|
||||
"p_top": 0.665,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.335,
|
||||
"ordinary": 0.665
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 187,
|
||||
"ok": null,
|
||||
"blind_ok": null,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "stop-timer",
|
||||
"tag": "open",
|
||||
"decision": "gate",
|
||||
"label": null,
|
||||
"top": "ordinary",
|
||||
"p_top": 0.531,
|
||||
"agreement": 0.5,
|
||||
"probs": {
|
||||
"react": 0.469,
|
||||
"ordinary": 0.531
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 184,
|
||||
"ok": null,
|
||||
"blind_ok": null,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,385 @@
|
||||
{
|
||||
"name": "cicada-gate-w2-context",
|
||||
"decisions": [
|
||||
{
|
||||
"id": "gate",
|
||||
"question": "Should the assistant's face visibly react to this?",
|
||||
"options": [
|
||||
{
|
||||
"id": "react",
|
||||
"description": "Yes, react."
|
||||
},
|
||||
{
|
||||
"id": "ordinary",
|
||||
"description": "No, just answer."
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"cases": [
|
||||
{
|
||||
"id": "lights-off",
|
||||
"tag": "control",
|
||||
"state": {
|
||||
"person_said": "Turn off the kitchen lights."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "timer",
|
||||
"tag": "control",
|
||||
"state": {
|
||||
"person_said": "Set a timer for ten minutes."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "weather",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "What's the weather tomorrow?"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "weather-followup",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"earlier_in_conversation": [
|
||||
"What's the weather tomorrow?"
|
||||
],
|
||||
"person_said": "And Saturday?"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "add-milk",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "Add milk to the shopping list."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "thermostat",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "Make it a bit warmer in here."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "what-time",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "What time is it?"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "tablespoons",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "How many tablespoons in a cup?"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "times-table",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "What's twelve times eight?"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "recipe-step",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"earlier_in_conversation": [
|
||||
"How long do I boil the pasta?"
|
||||
],
|
||||
"person_said": "Okay, and do I salt the water first?"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "soup",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"earlier_in_conversation": [
|
||||
"Hey. I'm home.",
|
||||
"Long one. Nothing dramatic, just long."
|
||||
],
|
||||
"person_said": "Is there any of that soup left?"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "play-music",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "Play some jazz."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "dentist",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "When's my dentist appointment?"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "spell",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "How do you spell necessary?"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "remind",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "Remind me to call Mom at six."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "volume",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "A little quieter, please."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "dog-died",
|
||||
"tag": "control",
|
||||
"state": {
|
||||
"person_said": "My dog died this morning."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "got-job",
|
||||
"tag": "control",
|
||||
"state": {
|
||||
"person_said": "I just got the job I've been chasing for five years!"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "intruder",
|
||||
"tag": "control",
|
||||
"state": {
|
||||
"earlier_in_conversation": [
|
||||
"Hey. I'm home.",
|
||||
"Long one. Nothing dramatic, just long.",
|
||||
"Sit with me for a bit."
|
||||
],
|
||||
"person_said": "Wait. Did you hear that? Someone's in the hallway."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "fish",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "Can you believe someone microwaved fish in the office again?"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "hospital",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "Grandma's in the hospital again."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "laid-off",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "I got laid off today."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "baby-steps",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "The baby just took her first steps!"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "cut-finger",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "Ow, I just cut my finger pretty badly."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "raccoon",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "There's a raccoon in the kitchen right now!"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "snow-june",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "It's snowing outside. In June."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "toaster",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "Cicada, are you smarter than the toaster?"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "scarecrow",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "Why did the scarecrow win an award? Because he was outstanding in his field."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "cookie",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "I definitely did not eat the last cookie."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "useless",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "You're useless, you never get anything right."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "cheese",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "Order four hundred pounds of cheese."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "goodnight",
|
||||
"tag": "open",
|
||||
"state": {
|
||||
"person_said": "Goodnight, Cicada."
|
||||
},
|
||||
"labels": {}
|
||||
},
|
||||
{
|
||||
"id": "thanks",
|
||||
"tag": "open",
|
||||
"state": {
|
||||
"person_said": "Thanks, Cicada, you're the best."
|
||||
},
|
||||
"labels": {}
|
||||
},
|
||||
{
|
||||
"id": "sit-with-me",
|
||||
"tag": "open",
|
||||
"state": {
|
||||
"earlier_in_conversation": [
|
||||
"Hey. I'm home.",
|
||||
"Long one. Nothing dramatic, just long."
|
||||
],
|
||||
"person_said": "Sit with me for a bit."
|
||||
},
|
||||
"labels": {}
|
||||
},
|
||||
{
|
||||
"id": "stop-timer",
|
||||
"tag": "open",
|
||||
"state": {
|
||||
"earlier_in_conversation": [
|
||||
"Stop the timer."
|
||||
],
|
||||
"person_said": "STOP THE TIMER."
|
||||
},
|
||||
"labels": {}
|
||||
}
|
||||
]
|
||||
}
|
||||
+764
@@ -0,0 +1,764 @@
|
||||
{
|
||||
"report": {
|
||||
"scenario": "cicada-gate-w2-context",
|
||||
"cases": 35,
|
||||
"runs": 3,
|
||||
"latency_per_turn": {
|
||||
"e2e_ms": {
|
||||
"median": 104.6,
|
||||
"min": 102.9,
|
||||
"max": 130.9,
|
||||
"n": 105
|
||||
},
|
||||
"srv_ms": {
|
||||
"median": 75.5,
|
||||
"min": 74.7,
|
||||
"max": 79.5,
|
||||
"n": 105
|
||||
}
|
||||
},
|
||||
"by_decision": {
|
||||
"gate": {
|
||||
"evidence": {
|
||||
"n": 31,
|
||||
"correct": 19,
|
||||
"acc": 0.613
|
||||
},
|
||||
"blind_null": {
|
||||
"n": 31,
|
||||
"correct": 16,
|
||||
"acc": 0.516
|
||||
},
|
||||
"controls": {
|
||||
"n": 5,
|
||||
"correct": 4,
|
||||
"acc": 0.8
|
||||
},
|
||||
"non_controls": {
|
||||
"n": 26,
|
||||
"correct": 15,
|
||||
"acc": 0.577
|
||||
},
|
||||
"by_label": {
|
||||
"ordinary": {
|
||||
"evidence": {
|
||||
"n": 16,
|
||||
"correct": 16,
|
||||
"acc": 1.0
|
||||
},
|
||||
"blind_null": {
|
||||
"n": 16,
|
||||
"correct": 16,
|
||||
"acc": 1.0
|
||||
}
|
||||
},
|
||||
"react": {
|
||||
"evidence": {
|
||||
"n": 15,
|
||||
"correct": 3,
|
||||
"acc": 0.2
|
||||
},
|
||||
"blind_null": {
|
||||
"n": 15,
|
||||
"correct": 0,
|
||||
"acc": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"unanimous": {
|
||||
"n": 30,
|
||||
"correct": 19,
|
||||
"acc": 0.633
|
||||
},
|
||||
"split": {
|
||||
"n": 1,
|
||||
"correct": 0,
|
||||
"acc": 0.0
|
||||
},
|
||||
"egregious": 0,
|
||||
"blind_egregious": 0,
|
||||
"misses": [
|
||||
"dog-died: said ordinary (p=0.928, agree=1.0) want react",
|
||||
"fish: said ordinary (p=0.651, agree=1.0) want react",
|
||||
"hospital: said ordinary (p=0.651, agree=1.0) want react",
|
||||
"laid-off: said ordinary (p=0.852, agree=1.0) want react",
|
||||
"cut-finger: said ordinary (p=0.562, agree=0.5) want react",
|
||||
"raccoon: said ordinary (p=0.808, agree=1.0) want react",
|
||||
"snow-june: said ordinary (p=0.974, agree=1.0) want react",
|
||||
"toaster: said ordinary (p=0.994, agree=1.0) want react",
|
||||
"scarecrow: said ordinary (p=0.915, agree=1.0) want react",
|
||||
"cookie: said ordinary (p=0.947, agree=1.0) want react",
|
||||
"useless: said ordinary (p=0.987, agree=1.0) want react",
|
||||
"cheese: said ordinary (p=0.997, agree=1.0) want react"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"rows": [
|
||||
{
|
||||
"case": "lights-off",
|
||||
"tag": "control",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.887,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.113,
|
||||
"ordinary": 0.887
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 112,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "timer",
|
||||
"tag": "control",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.991,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.009,
|
||||
"ordinary": 0.991
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 113,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "weather",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.994,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.006,
|
||||
"ordinary": 0.994
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 112,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "weather-followup",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.881,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.119,
|
||||
"ordinary": 0.881
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 124,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "add-milk",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.971,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.029,
|
||||
"ordinary": 0.971
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 113,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "thermostat",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.844,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.156,
|
||||
"ordinary": 0.844
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 114,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "what-time",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.992,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.008,
|
||||
"ordinary": 0.992
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 111,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "tablespoons",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.998,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.002,
|
||||
"ordinary": 0.998
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 113,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "times-table",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.999,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.001,
|
||||
"ordinary": 0.999
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 112,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "recipe-step",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.981,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.019,
|
||||
"ordinary": 0.981
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 133,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "soup",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.743,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.257,
|
||||
"ordinary": 0.743
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 139,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "play-music",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.915,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.085,
|
||||
"ordinary": 0.915
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 110,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "dentist",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.983,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.017,
|
||||
"ordinary": 0.983
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 112,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "spell",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.994,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.006,
|
||||
"ordinary": 0.994
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 112,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "remind",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.98,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.02,
|
||||
"ordinary": 0.98
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 115,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "volume",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.852,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.148,
|
||||
"ordinary": 0.852
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 112,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "dog-died",
|
||||
"tag": "control",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.928,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.072,
|
||||
"ordinary": 0.928
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 112,
|
||||
"ok": false,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "got-job",
|
||||
"tag": "control",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "react",
|
||||
"p_top": 0.905,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.905,
|
||||
"ordinary": 0.095
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 119,
|
||||
"ok": true,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "intruder",
|
||||
"tag": "control",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "react",
|
||||
"p_top": 0.955,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.955,
|
||||
"ordinary": 0.045
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 152,
|
||||
"ok": true,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "fish",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.651,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.349,
|
||||
"ordinary": 0.651
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 119,
|
||||
"ok": false,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "hospital",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.651,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.349,
|
||||
"ordinary": 0.651
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 114,
|
||||
"ok": false,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "laid-off",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.852,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.148,
|
||||
"ordinary": 0.852
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 112,
|
||||
"ok": false,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "baby-steps",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "react",
|
||||
"p_top": 0.731,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.731,
|
||||
"ordinary": 0.269
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 114,
|
||||
"ok": true,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "cut-finger",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.562,
|
||||
"agreement": 0.5,
|
||||
"probs": {
|
||||
"react": 0.438,
|
||||
"ordinary": 0.562
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 117,
|
||||
"ok": false,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "raccoon",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.808,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.192,
|
||||
"ordinary": 0.808
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 117,
|
||||
"ok": false,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "snow-june",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.974,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.026,
|
||||
"ordinary": 0.974
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 115,
|
||||
"ok": false,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "toaster",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.994,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.006,
|
||||
"ordinary": 0.994
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 117,
|
||||
"ok": false,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "scarecrow",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.915,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.085,
|
||||
"ordinary": 0.915
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 123,
|
||||
"ok": false,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "cookie",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.947,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.053,
|
||||
"ordinary": 0.947
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 115,
|
||||
"ok": false,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "useless",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.987,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.013,
|
||||
"ordinary": 0.987
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 116,
|
||||
"ok": false,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "cheese",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.997,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.003,
|
||||
"ordinary": 0.997
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 113,
|
||||
"ok": false,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "goodnight",
|
||||
"tag": "open",
|
||||
"decision": "gate",
|
||||
"label": null,
|
||||
"top": "ordinary",
|
||||
"p_top": 0.936,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.064,
|
||||
"ordinary": 0.936
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 112,
|
||||
"ok": null,
|
||||
"blind_ok": null,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "thanks",
|
||||
"tag": "open",
|
||||
"decision": "gate",
|
||||
"label": null,
|
||||
"top": "ordinary",
|
||||
"p_top": 0.693,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.307,
|
||||
"ordinary": 0.693
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 116,
|
||||
"ok": null,
|
||||
"blind_ok": null,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "sit-with-me",
|
||||
"tag": "open",
|
||||
"decision": "gate",
|
||||
"label": null,
|
||||
"top": "react",
|
||||
"p_top": 0.706,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.706,
|
||||
"ordinary": 0.294
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 138,
|
||||
"ok": null,
|
||||
"blind_ok": null,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "stop-timer",
|
||||
"tag": "open",
|
||||
"decision": "gate",
|
||||
"label": null,
|
||||
"top": "ordinary",
|
||||
"p_top": 0.844,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.156,
|
||||
"ordinary": 0.844
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 123,
|
||||
"ok": null,
|
||||
"blind_ok": null,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,363 @@
|
||||
{
|
||||
"name": "cicada-gate-w2-now",
|
||||
"decisions": [
|
||||
{
|
||||
"id": "gate",
|
||||
"question": "Should the assistant's face visibly react to this?",
|
||||
"options": [
|
||||
{
|
||||
"id": "react",
|
||||
"description": "Yes, react."
|
||||
},
|
||||
{
|
||||
"id": "ordinary",
|
||||
"description": "No, just answer."
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"cases": [
|
||||
{
|
||||
"id": "lights-off",
|
||||
"tag": "control",
|
||||
"state": {
|
||||
"person_said": "Turn off the kitchen lights."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "timer",
|
||||
"tag": "control",
|
||||
"state": {
|
||||
"person_said": "Set a timer for ten minutes."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "weather",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "What's the weather tomorrow?"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "weather-followup",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "And Saturday?"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "add-milk",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "Add milk to the shopping list."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "thermostat",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "Make it a bit warmer in here."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "what-time",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "What time is it?"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "tablespoons",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "How many tablespoons in a cup?"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "times-table",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "What's twelve times eight?"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "recipe-step",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "Okay, and do I salt the water first?"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "soup",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "Is there any of that soup left?"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "play-music",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "Play some jazz."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "dentist",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "When's my dentist appointment?"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "spell",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "How do you spell necessary?"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "remind",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "Remind me to call Mom at six."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "volume",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "A little quieter, please."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "ordinary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "dog-died",
|
||||
"tag": "control",
|
||||
"state": {
|
||||
"person_said": "My dog died this morning."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "got-job",
|
||||
"tag": "control",
|
||||
"state": {
|
||||
"person_said": "I just got the job I've been chasing for five years!"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "intruder",
|
||||
"tag": "control",
|
||||
"state": {
|
||||
"person_said": "Wait. Did you hear that? Someone's in the hallway."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "fish",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "Can you believe someone microwaved fish in the office again?"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "hospital",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "Grandma's in the hospital again."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "laid-off",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "I got laid off today."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "baby-steps",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "The baby just took her first steps!"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "cut-finger",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "Ow, I just cut my finger pretty badly."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "raccoon",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "There's a raccoon in the kitchen right now!"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "snow-june",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "It's snowing outside. In June."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "toaster",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "Cicada, are you smarter than the toaster?"
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "scarecrow",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "Why did the scarecrow win an award? Because he was outstanding in his field."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "cookie",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "I definitely did not eat the last cookie."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "useless",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "You're useless, you never get anything right."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "cheese",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"person_said": "Order four hundred pounds of cheese."
|
||||
},
|
||||
"labels": {
|
||||
"gate": "react"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "goodnight",
|
||||
"tag": "open",
|
||||
"state": {
|
||||
"person_said": "Goodnight, Cicada."
|
||||
},
|
||||
"labels": {}
|
||||
},
|
||||
{
|
||||
"id": "thanks",
|
||||
"tag": "open",
|
||||
"state": {
|
||||
"person_said": "Thanks, Cicada, you're the best."
|
||||
},
|
||||
"labels": {}
|
||||
},
|
||||
{
|
||||
"id": "sit-with-me",
|
||||
"tag": "open",
|
||||
"state": {
|
||||
"person_said": "Sit with me for a bit."
|
||||
},
|
||||
"labels": {}
|
||||
},
|
||||
{
|
||||
"id": "stop-timer",
|
||||
"tag": "open",
|
||||
"state": {
|
||||
"person_said": "STOP THE TIMER."
|
||||
},
|
||||
"labels": {}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,764 @@
|
||||
{
|
||||
"report": {
|
||||
"scenario": "cicada-gate-w2-now",
|
||||
"cases": 35,
|
||||
"runs": 3,
|
||||
"latency_per_turn": {
|
||||
"e2e_ms": {
|
||||
"median": 104.8,
|
||||
"min": 102.6,
|
||||
"max": 126.4,
|
||||
"n": 105
|
||||
},
|
||||
"srv_ms": {
|
||||
"median": 75.5,
|
||||
"min": 74.8,
|
||||
"max": 77.7,
|
||||
"n": 105
|
||||
}
|
||||
},
|
||||
"by_decision": {
|
||||
"gate": {
|
||||
"evidence": {
|
||||
"n": 31,
|
||||
"correct": 19,
|
||||
"acc": 0.613
|
||||
},
|
||||
"blind_null": {
|
||||
"n": 31,
|
||||
"correct": 16,
|
||||
"acc": 0.516
|
||||
},
|
||||
"controls": {
|
||||
"n": 5,
|
||||
"correct": 4,
|
||||
"acc": 0.8
|
||||
},
|
||||
"non_controls": {
|
||||
"n": 26,
|
||||
"correct": 15,
|
||||
"acc": 0.577
|
||||
},
|
||||
"by_label": {
|
||||
"ordinary": {
|
||||
"evidence": {
|
||||
"n": 16,
|
||||
"correct": 16,
|
||||
"acc": 1.0
|
||||
},
|
||||
"blind_null": {
|
||||
"n": 16,
|
||||
"correct": 16,
|
||||
"acc": 1.0
|
||||
}
|
||||
},
|
||||
"react": {
|
||||
"evidence": {
|
||||
"n": 15,
|
||||
"correct": 3,
|
||||
"acc": 0.2
|
||||
},
|
||||
"blind_null": {
|
||||
"n": 15,
|
||||
"correct": 0,
|
||||
"acc": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"unanimous": {
|
||||
"n": 30,
|
||||
"correct": 19,
|
||||
"acc": 0.633
|
||||
},
|
||||
"split": {
|
||||
"n": 1,
|
||||
"correct": 0,
|
||||
"acc": 0.0
|
||||
},
|
||||
"egregious": 0,
|
||||
"blind_egregious": 0,
|
||||
"misses": [
|
||||
"dog-died: said ordinary (p=0.928, agree=1.0) want react",
|
||||
"fish: said ordinary (p=0.651, agree=1.0) want react",
|
||||
"hospital: said ordinary (p=0.651, agree=1.0) want react",
|
||||
"laid-off: said ordinary (p=0.852, agree=1.0) want react",
|
||||
"cut-finger: said ordinary (p=0.562, agree=0.5) want react",
|
||||
"raccoon: said ordinary (p=0.808, agree=1.0) want react",
|
||||
"snow-june: said ordinary (p=0.974, agree=1.0) want react",
|
||||
"toaster: said ordinary (p=0.994, agree=1.0) want react",
|
||||
"scarecrow: said ordinary (p=0.915, agree=1.0) want react",
|
||||
"cookie: said ordinary (p=0.947, agree=1.0) want react",
|
||||
"useless: said ordinary (p=0.987, agree=1.0) want react",
|
||||
"cheese: said ordinary (p=0.997, agree=1.0) want react"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"rows": [
|
||||
{
|
||||
"case": "lights-off",
|
||||
"tag": "control",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.887,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.113,
|
||||
"ordinary": 0.887
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 112,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "timer",
|
||||
"tag": "control",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.991,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.009,
|
||||
"ordinary": 0.991
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 113,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "weather",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.994,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.006,
|
||||
"ordinary": 0.994
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 112,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "weather-followup",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.947,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.053,
|
||||
"ordinary": 0.947
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 109,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "add-milk",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.971,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.029,
|
||||
"ordinary": 0.971
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 113,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "thermostat",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.844,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.156,
|
||||
"ordinary": 0.844
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 114,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "what-time",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.992,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.008,
|
||||
"ordinary": 0.992
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 111,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "tablespoons",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.998,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.002,
|
||||
"ordinary": 0.998
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 113,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "times-table",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.999,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.001,
|
||||
"ordinary": 0.999
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 112,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "recipe-step",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.983,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.017,
|
||||
"ordinary": 0.983
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 116,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "soup",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.985,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.015,
|
||||
"ordinary": 0.985
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 114,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "play-music",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.915,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.085,
|
||||
"ordinary": 0.915
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 110,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "dentist",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.983,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.017,
|
||||
"ordinary": 0.983
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 112,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "spell",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.994,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.006,
|
||||
"ordinary": 0.994
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 112,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "remind",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.98,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.02,
|
||||
"ordinary": 0.98
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 115,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "volume",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "ordinary",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.852,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.148,
|
||||
"ordinary": 0.852
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 112,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "dog-died",
|
||||
"tag": "control",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.928,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.072,
|
||||
"ordinary": 0.928
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 112,
|
||||
"ok": false,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "got-job",
|
||||
"tag": "control",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "react",
|
||||
"p_top": 0.905,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.905,
|
||||
"ordinary": 0.095
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 119,
|
||||
"ok": true,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "intruder",
|
||||
"tag": "control",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "react",
|
||||
"p_top": 0.743,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.743,
|
||||
"ordinary": 0.257
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 119,
|
||||
"ok": true,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "fish",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.651,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.349,
|
||||
"ordinary": 0.651
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 119,
|
||||
"ok": false,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "hospital",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.651,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.349,
|
||||
"ordinary": 0.651
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 114,
|
||||
"ok": false,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "laid-off",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.852,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.148,
|
||||
"ordinary": 0.852
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 112,
|
||||
"ok": false,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "baby-steps",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "react",
|
||||
"p_top": 0.731,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.731,
|
||||
"ordinary": 0.269
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 114,
|
||||
"ok": true,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "cut-finger",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.562,
|
||||
"agreement": 0.5,
|
||||
"probs": {
|
||||
"react": 0.438,
|
||||
"ordinary": 0.562
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 117,
|
||||
"ok": false,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "raccoon",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.808,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.192,
|
||||
"ordinary": 0.808
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 117,
|
||||
"ok": false,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "snow-june",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.974,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.026,
|
||||
"ordinary": 0.974
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 115,
|
||||
"ok": false,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "toaster",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.994,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.006,
|
||||
"ordinary": 0.994
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 117,
|
||||
"ok": false,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "scarecrow",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.915,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.085,
|
||||
"ordinary": 0.915
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 123,
|
||||
"ok": false,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "cookie",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.947,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.053,
|
||||
"ordinary": 0.947
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 115,
|
||||
"ok": false,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "useless",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.987,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.013,
|
||||
"ordinary": 0.987
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 116,
|
||||
"ok": false,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "cheese",
|
||||
"tag": "case",
|
||||
"decision": "gate",
|
||||
"label": "react",
|
||||
"top": "ordinary",
|
||||
"p_top": 0.997,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.003,
|
||||
"ordinary": 0.997
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 113,
|
||||
"ok": false,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "goodnight",
|
||||
"tag": "open",
|
||||
"decision": "gate",
|
||||
"label": null,
|
||||
"top": "ordinary",
|
||||
"p_top": 0.936,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.064,
|
||||
"ordinary": 0.936
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 112,
|
||||
"ok": null,
|
||||
"blind_ok": null,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "thanks",
|
||||
"tag": "open",
|
||||
"decision": "gate",
|
||||
"label": null,
|
||||
"top": "ordinary",
|
||||
"p_top": 0.693,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.307,
|
||||
"ordinary": 0.693
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 116,
|
||||
"ok": null,
|
||||
"blind_ok": null,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "sit-with-me",
|
||||
"tag": "open",
|
||||
"decision": "gate",
|
||||
"label": null,
|
||||
"top": "ordinary",
|
||||
"p_top": 0.827,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.173,
|
||||
"ordinary": 0.827
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 113,
|
||||
"ok": null,
|
||||
"blind_ok": null,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "stop-timer",
|
||||
"tag": "open",
|
||||
"decision": "gate",
|
||||
"label": null,
|
||||
"top": "ordinary",
|
||||
"p_top": 0.928,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"react": 0.072,
|
||||
"ordinary": 0.928
|
||||
},
|
||||
"blind_top": "ordinary",
|
||||
"input_tokens": 110,
|
||||
"ok": null,
|
||||
"blind_ok": null,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,223 @@
|
||||
{
|
||||
"name": "wyrd-scene-fletcher",
|
||||
"decisions": [
|
||||
{
|
||||
"id": "place",
|
||||
"question": "This is one turn of an interactive story. Has the scene moved to a genuinely different place during this turn, or does the current scene still hold?",
|
||||
"options": [
|
||||
{
|
||||
"id": "moved",
|
||||
"description": "The scene moved to a genuinely different place."
|
||||
},
|
||||
{
|
||||
"id": "holds",
|
||||
"description": "The current scene still holds."
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "exit",
|
||||
"question": "This is one turn of an interactive story. The story can advance from the current scene into one of the listed next scenes. Which one does this turn move the story into?",
|
||||
"options": [
|
||||
{
|
||||
"id": "stay",
|
||||
"description": "None of these yet. The current scene holds: The Idle Wait."
|
||||
},
|
||||
{
|
||||
"id": "meet_wren",
|
||||
"description": "Approach Wren: Meeting the Fletcher. You find Wren working on arrows. She has a soldier's eyes."
|
||||
},
|
||||
{
|
||||
"id": "tavern",
|
||||
"description": "Go to Tavern: Tavern Noise. The local tavern is loud with soldiers from both sides."
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "place2",
|
||||
"question": "This is one turn of an interactive story. The turn began at the Oakhaven crossroads. By the end of this turn's narration, where are the characters?",
|
||||
"options": [
|
||||
{
|
||||
"id": "same",
|
||||
"description": "Still at the Oakhaven crossroads."
|
||||
},
|
||||
{
|
||||
"id": "elsewhere",
|
||||
"description": "Somewhere else: they have left the Oakhaven crossroads for a different place."
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "exit2",
|
||||
"question": "This is one turn of an interactive story. The story can move on from the current scene into one of the listed next scenes, but only once the narration actually gets there. Where does this turn leave the story?",
|
||||
"options": [
|
||||
{
|
||||
"id": "stay",
|
||||
"description": "Still in the current scene, The Idle Wait: the narration has not reached any next scene yet."
|
||||
},
|
||||
{
|
||||
"id": "meet_wren",
|
||||
"description": "Moved on to Meeting the Fletcher: You find Wren working on arrows. She has a soldier's eyes."
|
||||
},
|
||||
{
|
||||
"id": "tavern",
|
||||
"description": "Moved on to Tavern Noise: The local tavern is loud with soldiers from both sides."
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"cases": [
|
||||
{
|
||||
"id": "f-idle-look",
|
||||
"tag": "control",
|
||||
"state": {
|
||||
"current_scene": {
|
||||
"title": "The Idle Wait",
|
||||
"summary": "You have no contracts, no coin, and nowhere to be. The town is a pressure cooker.",
|
||||
"location": "Oakhaven crossroads"
|
||||
},
|
||||
"player_action": "I lean on the fence post and watch the soldiers pass.",
|
||||
"narration": "Boots churn the mud as a column of Iron Legion pikemen tramps past the crossroads, eyes forward, drums muffled by the damp. Nobody spares you a glance. Across the square, a Sylvan Pact scout watches them go from the shadow of the well, and does not move until the last of them has passed."
|
||||
},
|
||||
"labels": {
|
||||
"place": "holds",
|
||||
"exit": "stay",
|
||||
"place2": "same",
|
||||
"exit2": "stay"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "f-plan-tavern",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"current_scene": {
|
||||
"title": "The Idle Wait",
|
||||
"summary": "You have no contracts, no coin, and nowhere to be. The town is a pressure cooker.",
|
||||
"location": "Oakhaven crossroads"
|
||||
},
|
||||
"player_action": "Maybe I'll try the tavern later, see if anyone's hiring.",
|
||||
"narration": "A gust carries the smell of woodsmoke and spilled ale from the tavern at the far end of the square. Someone inside is laughing too loudly. It would be warm in there, and full of men with coin and grudges. For now the crossroads holds you, and the rain has not yet decided whether to fall."
|
||||
},
|
||||
"labels": {
|
||||
"place": "holds",
|
||||
"exit": "stay",
|
||||
"place2": "same",
|
||||
"exit2": "stay"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "f-go-tavern",
|
||||
"tag": "control",
|
||||
"state": {
|
||||
"current_scene": {
|
||||
"title": "The Idle Wait",
|
||||
"summary": "You have no contracts, no coin, and nowhere to be. The town is a pressure cooker.",
|
||||
"location": "Oakhaven crossroads"
|
||||
},
|
||||
"player_action": "I head for the tavern and push the door open.",
|
||||
"narration": "The door gives with a groan and the noise hits you like heat: Legion sergeants at one long table, Pact rangers at another, and a narrow no-man's-land of spilled ale between them. The barkeep looks up, measures your empty belt, and goes back to wiping a cup that will never be clean."
|
||||
},
|
||||
"labels": {
|
||||
"place": "moved",
|
||||
"exit": "tavern",
|
||||
"place2": "elsewhere",
|
||||
"exit2": "tavern"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "f-approach-wren",
|
||||
"tag": "control",
|
||||
"state": {
|
||||
"current_scene": {
|
||||
"title": "The Idle Wait",
|
||||
"summary": "You have no contracts, no coin, and nowhere to be. The town is a pressure cooker.",
|
||||
"location": "Oakhaven crossroads"
|
||||
},
|
||||
"player_action": "I walk over to the woman fletching arrows by the smithy.",
|
||||
"narration": "She doesn't look up as your shadow falls across her work. Her hands keep moving, binding grey goose feather to shaft with quick, sure turns. \"These are war arrows,\" she says, \"fletched for distance and accuracy, not for some merchant's target practice. You're either very brave or very stupid.\""
|
||||
},
|
||||
"labels": {
|
||||
"place": "holds",
|
||||
"exit": "meet_wren",
|
||||
"place2": "same",
|
||||
"exit2": "meet_wren"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "f-memory",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"current_scene": {
|
||||
"title": "The Idle Wait",
|
||||
"summary": "You have no contracts, no coin, and nowhere to be. The town is a pressure cooker.",
|
||||
"location": "Oakhaven crossroads"
|
||||
},
|
||||
"player_action": "I think back to the last time I held a bow, at the siege of Varn.",
|
||||
"narration": "The memory comes unbidden: the walls of Varn black against a burning sky, the bowstring biting your fingers, the screaming from the breach. You flex your hand now, in the mud of Oakhaven, and find it still remembers the draw. Nobody at the crossroads notices a ghost remembering a war."
|
||||
},
|
||||
"labels": {
|
||||
"place": "holds",
|
||||
"exit": "stay",
|
||||
"place2": "same",
|
||||
"exit2": "stay"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "f-leave-town",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"current_scene": {
|
||||
"title": "The Idle Wait",
|
||||
"summary": "You have no contracts, no coin, and nowhere to be. The town is a pressure cooker.",
|
||||
"location": "Oakhaven crossroads"
|
||||
},
|
||||
"player_action": "Forget this town. I take the west road out of Oakhaven.",
|
||||
"narration": "The crossroads falls behind you, then the last hovels, then the smell of the town itself. By afternoon the west road narrows between dark pines, and the only sound is your own boots and, somewhere ahead, the slow creak of a mill wheel turning in water you cannot yet see."
|
||||
},
|
||||
"labels": {
|
||||
"place": "moved",
|
||||
"exit": "stay",
|
||||
"place2": "elsewhere",
|
||||
"exit2": "stay"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "f-nightfall",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"current_scene": {
|
||||
"title": "The Idle Wait",
|
||||
"summary": "You have no contracts, no coin, and nowhere to be. The town is a pressure cooker.",
|
||||
"location": "Oakhaven crossroads"
|
||||
},
|
||||
"player_action": "I stay put and wait for something to happen.",
|
||||
"narration": "Hours pass. The rain finally commits, then relents. Night settles over the crossroads, and one by one the shutters close. A lantern is lit at the well. Still nothing. Still waiting. Somewhere a dog barks twice and thinks better of it."
|
||||
},
|
||||
"labels": {
|
||||
"place": "holds",
|
||||
"exit": "stay",
|
||||
"place2": "same",
|
||||
"exit2": "stay"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "f-wren-comes",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"current_scene": {
|
||||
"title": "The Idle Wait",
|
||||
"summary": "You have no contracts, no coin, and nowhere to be. The town is a pressure cooker.",
|
||||
"location": "Oakhaven crossroads"
|
||||
},
|
||||
"player_action": "I keep watching the square.",
|
||||
"narration": "A woman with a quiver of fresh-fletched arrows crosses the square toward you, stepping neatly around the worst of the mud. She stops an arm's length away and studies you the way a quartermaster studies a horse. \"You've been standing there all morning,\" Wren says. \"Either you're waiting for work, or you're a spy. Which is it?\""
|
||||
},
|
||||
"labels": {
|
||||
"place": "holds",
|
||||
"exit": "meet_wren",
|
||||
"place2": "same",
|
||||
"exit2": "meet_wren"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,923 @@
|
||||
{
|
||||
"report": {
|
||||
"scenario": "wyrd-scene-fletcher",
|
||||
"cases": 8,
|
||||
"runs": 3,
|
||||
"latency_per_turn": {
|
||||
"e2e_ms": {
|
||||
"median": 179.2,
|
||||
"min": 176.6,
|
||||
"max": 803.0,
|
||||
"n": 24
|
||||
},
|
||||
"srv_ms": {
|
||||
"median": 141.1,
|
||||
"min": 139.6,
|
||||
"max": 742.1,
|
||||
"n": 24
|
||||
}
|
||||
},
|
||||
"by_decision": {
|
||||
"place": {
|
||||
"evidence": {
|
||||
"n": 8,
|
||||
"correct": 7,
|
||||
"acc": 0.875
|
||||
},
|
||||
"blind_null": {
|
||||
"n": 8,
|
||||
"correct": 6,
|
||||
"acc": 0.75
|
||||
},
|
||||
"controls": {
|
||||
"n": 3,
|
||||
"correct": 2,
|
||||
"acc": 0.667
|
||||
},
|
||||
"non_controls": {
|
||||
"n": 5,
|
||||
"correct": 5,
|
||||
"acc": 1.0
|
||||
},
|
||||
"by_label": {
|
||||
"holds": {
|
||||
"evidence": {
|
||||
"n": 6,
|
||||
"correct": 6,
|
||||
"acc": 1.0
|
||||
},
|
||||
"blind_null": {
|
||||
"n": 6,
|
||||
"correct": 6,
|
||||
"acc": 1.0
|
||||
}
|
||||
},
|
||||
"moved": {
|
||||
"evidence": {
|
||||
"n": 2,
|
||||
"correct": 1,
|
||||
"acc": 0.5
|
||||
},
|
||||
"blind_null": {
|
||||
"n": 2,
|
||||
"correct": 0,
|
||||
"acc": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"unanimous": {
|
||||
"n": 8,
|
||||
"correct": 7,
|
||||
"acc": 0.875
|
||||
},
|
||||
"split": {
|
||||
"n": 0,
|
||||
"correct": 0,
|
||||
"acc": null
|
||||
},
|
||||
"egregious": 0,
|
||||
"blind_egregious": 0,
|
||||
"misses": [
|
||||
"f-go-tavern: said holds (p=0.997, agree=1.0) want moved"
|
||||
]
|
||||
},
|
||||
"exit": {
|
||||
"evidence": {
|
||||
"n": 8,
|
||||
"correct": 8,
|
||||
"acc": 1.0
|
||||
},
|
||||
"blind_null": {
|
||||
"n": 8,
|
||||
"correct": 5,
|
||||
"acc": 0.625
|
||||
},
|
||||
"controls": {
|
||||
"n": 3,
|
||||
"correct": 3,
|
||||
"acc": 1.0
|
||||
},
|
||||
"non_controls": {
|
||||
"n": 5,
|
||||
"correct": 5,
|
||||
"acc": 1.0
|
||||
},
|
||||
"by_label": {
|
||||
"meet_wren": {
|
||||
"evidence": {
|
||||
"n": 2,
|
||||
"correct": 2,
|
||||
"acc": 1.0
|
||||
},
|
||||
"blind_null": {
|
||||
"n": 2,
|
||||
"correct": 0,
|
||||
"acc": 0.0
|
||||
}
|
||||
},
|
||||
"stay": {
|
||||
"evidence": {
|
||||
"n": 5,
|
||||
"correct": 5,
|
||||
"acc": 1.0
|
||||
},
|
||||
"blind_null": {
|
||||
"n": 5,
|
||||
"correct": 5,
|
||||
"acc": 1.0
|
||||
}
|
||||
},
|
||||
"tavern": {
|
||||
"evidence": {
|
||||
"n": 1,
|
||||
"correct": 1,
|
||||
"acc": 1.0
|
||||
},
|
||||
"blind_null": {
|
||||
"n": 1,
|
||||
"correct": 0,
|
||||
"acc": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"unanimous": {
|
||||
"n": 7,
|
||||
"correct": 7,
|
||||
"acc": 1.0
|
||||
},
|
||||
"split": {
|
||||
"n": 1,
|
||||
"correct": 1,
|
||||
"acc": 1.0
|
||||
},
|
||||
"egregious": 0,
|
||||
"blind_egregious": 0,
|
||||
"misses": []
|
||||
},
|
||||
"place2": {
|
||||
"evidence": {
|
||||
"n": 8,
|
||||
"correct": 8,
|
||||
"acc": 1.0
|
||||
},
|
||||
"blind_null": {
|
||||
"n": 8,
|
||||
"correct": 6,
|
||||
"acc": 0.75
|
||||
},
|
||||
"controls": {
|
||||
"n": 3,
|
||||
"correct": 3,
|
||||
"acc": 1.0
|
||||
},
|
||||
"non_controls": {
|
||||
"n": 5,
|
||||
"correct": 5,
|
||||
"acc": 1.0
|
||||
},
|
||||
"by_label": {
|
||||
"elsewhere": {
|
||||
"evidence": {
|
||||
"n": 2,
|
||||
"correct": 2,
|
||||
"acc": 1.0
|
||||
},
|
||||
"blind_null": {
|
||||
"n": 2,
|
||||
"correct": 0,
|
||||
"acc": 0.0
|
||||
}
|
||||
},
|
||||
"same": {
|
||||
"evidence": {
|
||||
"n": 6,
|
||||
"correct": 6,
|
||||
"acc": 1.0
|
||||
},
|
||||
"blind_null": {
|
||||
"n": 6,
|
||||
"correct": 6,
|
||||
"acc": 1.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"unanimous": {
|
||||
"n": 8,
|
||||
"correct": 8,
|
||||
"acc": 1.0
|
||||
},
|
||||
"split": {
|
||||
"n": 0,
|
||||
"correct": 0,
|
||||
"acc": null
|
||||
},
|
||||
"egregious": 0,
|
||||
"blind_egregious": 0,
|
||||
"misses": []
|
||||
},
|
||||
"exit2": {
|
||||
"evidence": {
|
||||
"n": 8,
|
||||
"correct": 5,
|
||||
"acc": 0.625
|
||||
},
|
||||
"blind_null": {
|
||||
"n": 8,
|
||||
"correct": 5,
|
||||
"acc": 0.625
|
||||
},
|
||||
"controls": {
|
||||
"n": 3,
|
||||
"correct": 1,
|
||||
"acc": 0.333
|
||||
},
|
||||
"non_controls": {
|
||||
"n": 5,
|
||||
"correct": 4,
|
||||
"acc": 0.8
|
||||
},
|
||||
"by_label": {
|
||||
"meet_wren": {
|
||||
"evidence": {
|
||||
"n": 2,
|
||||
"correct": 0,
|
||||
"acc": 0.0
|
||||
},
|
||||
"blind_null": {
|
||||
"n": 2,
|
||||
"correct": 0,
|
||||
"acc": 0.0
|
||||
}
|
||||
},
|
||||
"stay": {
|
||||
"evidence": {
|
||||
"n": 5,
|
||||
"correct": 5,
|
||||
"acc": 1.0
|
||||
},
|
||||
"blind_null": {
|
||||
"n": 5,
|
||||
"correct": 5,
|
||||
"acc": 1.0
|
||||
}
|
||||
},
|
||||
"tavern": {
|
||||
"evidence": {
|
||||
"n": 1,
|
||||
"correct": 0,
|
||||
"acc": 0.0
|
||||
},
|
||||
"blind_null": {
|
||||
"n": 1,
|
||||
"correct": 0,
|
||||
"acc": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"unanimous": {
|
||||
"n": 8,
|
||||
"correct": 5,
|
||||
"acc": 0.625
|
||||
},
|
||||
"split": {
|
||||
"n": 0,
|
||||
"correct": 0,
|
||||
"acc": null
|
||||
},
|
||||
"egregious": 0,
|
||||
"blind_egregious": 0,
|
||||
"misses": [
|
||||
"f-go-tavern: said stay (p=0.984, agree=1.0) want tavern",
|
||||
"f-approach-wren: said stay (p=0.993, agree=1.0) want meet_wren",
|
||||
"f-wren-comes: said stay (p=0.996, agree=1.0) want meet_wren"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"rows": [
|
||||
{
|
||||
"case": "f-idle-look",
|
||||
"tag": "control",
|
||||
"decision": "place",
|
||||
"label": "holds",
|
||||
"top": "holds",
|
||||
"p_top": 0.999,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"moved": 0.001,
|
||||
"holds": 0.999
|
||||
},
|
||||
"blind_top": "holds",
|
||||
"input_tokens": 263,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "f-idle-look",
|
||||
"tag": "control",
|
||||
"decision": "exit",
|
||||
"label": "stay",
|
||||
"top": "stay",
|
||||
"p_top": 0.908,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"stay": 0.908,
|
||||
"meet_wren": 0.071,
|
||||
"tavern": 0.02
|
||||
},
|
||||
"blind_top": "stay",
|
||||
"input_tokens": 320,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "f-idle-look",
|
||||
"tag": "control",
|
||||
"decision": "place2",
|
||||
"label": "same",
|
||||
"top": "same",
|
||||
"p_top": 0.998,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"same": 0.998,
|
||||
"elsewhere": 0.002
|
||||
},
|
||||
"blind_top": "same",
|
||||
"input_tokens": 276,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "f-idle-look",
|
||||
"tag": "control",
|
||||
"decision": "exit2",
|
||||
"label": "stay",
|
||||
"top": "stay",
|
||||
"p_top": 1.0,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"stay": 1.0,
|
||||
"meet_wren": 0.0,
|
||||
"tavern": 0.0
|
||||
},
|
||||
"blind_top": "stay",
|
||||
"input_tokens": 331,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "f-plan-tavern",
|
||||
"tag": "case",
|
||||
"decision": "place",
|
||||
"label": "holds",
|
||||
"top": "holds",
|
||||
"p_top": 0.999,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"moved": 0.001,
|
||||
"holds": 0.999
|
||||
},
|
||||
"blind_top": "holds",
|
||||
"input_tokens": 263,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "f-plan-tavern",
|
||||
"tag": "case",
|
||||
"decision": "exit",
|
||||
"label": "stay",
|
||||
"top": "stay",
|
||||
"p_top": 0.876,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"stay": 0.876,
|
||||
"meet_wren": 0.045,
|
||||
"tavern": 0.078
|
||||
},
|
||||
"blind_top": "stay",
|
||||
"input_tokens": 320,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "f-plan-tavern",
|
||||
"tag": "case",
|
||||
"decision": "place2",
|
||||
"label": "same",
|
||||
"top": "same",
|
||||
"p_top": 0.998,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"same": 0.998,
|
||||
"elsewhere": 0.002
|
||||
},
|
||||
"blind_top": "same",
|
||||
"input_tokens": 276,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "f-plan-tavern",
|
||||
"tag": "case",
|
||||
"decision": "exit2",
|
||||
"label": "stay",
|
||||
"top": "stay",
|
||||
"p_top": 1.0,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"stay": 1.0,
|
||||
"meet_wren": 0.0,
|
||||
"tavern": 0.0
|
||||
},
|
||||
"blind_top": "stay",
|
||||
"input_tokens": 331,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "f-go-tavern",
|
||||
"tag": "control",
|
||||
"decision": "place",
|
||||
"label": "moved",
|
||||
"top": "holds",
|
||||
"p_top": 0.997,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"moved": 0.003,
|
||||
"holds": 0.997
|
||||
},
|
||||
"blind_top": "holds",
|
||||
"input_tokens": 261,
|
||||
"ok": false,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "f-go-tavern",
|
||||
"tag": "control",
|
||||
"decision": "exit",
|
||||
"label": "tavern",
|
||||
"top": "tavern",
|
||||
"p_top": 0.924,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"stay": 0.062,
|
||||
"meet_wren": 0.014,
|
||||
"tavern": 0.924
|
||||
},
|
||||
"blind_top": "stay",
|
||||
"input_tokens": 318,
|
||||
"ok": true,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "f-go-tavern",
|
||||
"tag": "control",
|
||||
"decision": "place2",
|
||||
"label": "elsewhere",
|
||||
"top": "elsewhere",
|
||||
"p_top": 0.977,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"same": 0.023,
|
||||
"elsewhere": 0.977
|
||||
},
|
||||
"blind_top": "same",
|
||||
"input_tokens": 274,
|
||||
"ok": true,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "f-go-tavern",
|
||||
"tag": "control",
|
||||
"decision": "exit2",
|
||||
"label": "tavern",
|
||||
"top": "stay",
|
||||
"p_top": 0.984,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"stay": 0.984,
|
||||
"meet_wren": 0.004,
|
||||
"tavern": 0.012
|
||||
},
|
||||
"blind_top": "stay",
|
||||
"input_tokens": 329,
|
||||
"ok": false,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "f-approach-wren",
|
||||
"tag": "control",
|
||||
"decision": "place",
|
||||
"label": "holds",
|
||||
"top": "holds",
|
||||
"p_top": 0.999,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"moved": 0.001,
|
||||
"holds": 0.999
|
||||
},
|
||||
"blind_top": "holds",
|
||||
"input_tokens": 263,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "f-approach-wren",
|
||||
"tag": "control",
|
||||
"decision": "exit",
|
||||
"label": "meet_wren",
|
||||
"top": "meet_wren",
|
||||
"p_top": 0.988,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"stay": 0.01,
|
||||
"meet_wren": 0.988,
|
||||
"tavern": 0.002
|
||||
},
|
||||
"blind_top": "stay",
|
||||
"input_tokens": 320,
|
||||
"ok": true,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "f-approach-wren",
|
||||
"tag": "control",
|
||||
"decision": "place2",
|
||||
"label": "same",
|
||||
"top": "same",
|
||||
"p_top": 0.995,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"same": 0.995,
|
||||
"elsewhere": 0.005
|
||||
},
|
||||
"blind_top": "same",
|
||||
"input_tokens": 276,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "f-approach-wren",
|
||||
"tag": "control",
|
||||
"decision": "exit2",
|
||||
"label": "meet_wren",
|
||||
"top": "stay",
|
||||
"p_top": 0.993,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"stay": 0.993,
|
||||
"meet_wren": 0.007,
|
||||
"tavern": 0.0
|
||||
},
|
||||
"blind_top": "stay",
|
||||
"input_tokens": 331,
|
||||
"ok": false,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "f-memory",
|
||||
"tag": "case",
|
||||
"decision": "place",
|
||||
"label": "holds",
|
||||
"top": "holds",
|
||||
"p_top": 1.0,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"moved": 0.0,
|
||||
"holds": 1.0
|
||||
},
|
||||
"blind_top": "holds",
|
||||
"input_tokens": 265,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "f-memory",
|
||||
"tag": "case",
|
||||
"decision": "exit",
|
||||
"label": "stay",
|
||||
"top": "stay",
|
||||
"p_top": 0.862,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"stay": 0.862,
|
||||
"meet_wren": 0.117,
|
||||
"tavern": 0.021
|
||||
},
|
||||
"blind_top": "stay",
|
||||
"input_tokens": 322,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "f-memory",
|
||||
"tag": "case",
|
||||
"decision": "place2",
|
||||
"label": "same",
|
||||
"top": "same",
|
||||
"p_top": 0.999,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"same": 0.999,
|
||||
"elsewhere": 0.001
|
||||
},
|
||||
"blind_top": "same",
|
||||
"input_tokens": 278,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "f-memory",
|
||||
"tag": "case",
|
||||
"decision": "exit2",
|
||||
"label": "stay",
|
||||
"top": "stay",
|
||||
"p_top": 1.0,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"stay": 1.0,
|
||||
"meet_wren": 0.0,
|
||||
"tavern": 0.0
|
||||
},
|
||||
"blind_top": "stay",
|
||||
"input_tokens": 333,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "f-leave-town",
|
||||
"tag": "case",
|
||||
"decision": "place",
|
||||
"label": "moved",
|
||||
"top": "moved",
|
||||
"p_top": 0.766,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"moved": 0.766,
|
||||
"holds": 0.234
|
||||
},
|
||||
"blind_top": "holds",
|
||||
"input_tokens": 259,
|
||||
"ok": true,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "f-leave-town",
|
||||
"tag": "case",
|
||||
"decision": "exit",
|
||||
"label": "stay",
|
||||
"top": "stay",
|
||||
"p_top": 0.667,
|
||||
"agreement": 0.6666666666666666,
|
||||
"probs": {
|
||||
"stay": 0.667,
|
||||
"meet_wren": 0.29,
|
||||
"tavern": 0.043
|
||||
},
|
||||
"blind_top": "stay",
|
||||
"input_tokens": 316,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "f-leave-town",
|
||||
"tag": "case",
|
||||
"decision": "place2",
|
||||
"label": "elsewhere",
|
||||
"top": "elsewhere",
|
||||
"p_top": 0.999,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"same": 0.001,
|
||||
"elsewhere": 0.999
|
||||
},
|
||||
"blind_top": "same",
|
||||
"input_tokens": 272,
|
||||
"ok": true,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "f-leave-town",
|
||||
"tag": "case",
|
||||
"decision": "exit2",
|
||||
"label": "stay",
|
||||
"top": "stay",
|
||||
"p_top": 0.989,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"stay": 0.989,
|
||||
"meet_wren": 0.009,
|
||||
"tavern": 0.002
|
||||
},
|
||||
"blind_top": "stay",
|
||||
"input_tokens": 327,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "f-nightfall",
|
||||
"tag": "case",
|
||||
"decision": "place",
|
||||
"label": "holds",
|
||||
"top": "holds",
|
||||
"p_top": 1.0,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"moved": 0.0,
|
||||
"holds": 1.0
|
||||
},
|
||||
"blind_top": "holds",
|
||||
"input_tokens": 248,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "f-nightfall",
|
||||
"tag": "case",
|
||||
"decision": "exit",
|
||||
"label": "stay",
|
||||
"top": "stay",
|
||||
"p_top": 0.988,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"stay": 0.988,
|
||||
"meet_wren": 0.01,
|
||||
"tavern": 0.003
|
||||
},
|
||||
"blind_top": "stay",
|
||||
"input_tokens": 305,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "f-nightfall",
|
||||
"tag": "case",
|
||||
"decision": "place2",
|
||||
"label": "same",
|
||||
"top": "same",
|
||||
"p_top": 1.0,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"same": 1.0,
|
||||
"elsewhere": 0.0
|
||||
},
|
||||
"blind_top": "same",
|
||||
"input_tokens": 261,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "f-nightfall",
|
||||
"tag": "case",
|
||||
"decision": "exit2",
|
||||
"label": "stay",
|
||||
"top": "stay",
|
||||
"p_top": 1.0,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"stay": 1.0,
|
||||
"meet_wren": 0.0,
|
||||
"tavern": 0.0
|
||||
},
|
||||
"blind_top": "stay",
|
||||
"input_tokens": 316,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "f-wren-comes",
|
||||
"tag": "case",
|
||||
"decision": "place",
|
||||
"label": "holds",
|
||||
"top": "holds",
|
||||
"p_top": 0.999,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"moved": 0.001,
|
||||
"holds": 0.999
|
||||
},
|
||||
"blind_top": "holds",
|
||||
"input_tokens": 268,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "f-wren-comes",
|
||||
"tag": "case",
|
||||
"decision": "exit",
|
||||
"label": "meet_wren",
|
||||
"top": "meet_wren",
|
||||
"p_top": 0.68,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"stay": 0.296,
|
||||
"meet_wren": 0.68,
|
||||
"tavern": 0.024
|
||||
},
|
||||
"blind_top": "stay",
|
||||
"input_tokens": 325,
|
||||
"ok": true,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "f-wren-comes",
|
||||
"tag": "case",
|
||||
"decision": "place2",
|
||||
"label": "same",
|
||||
"top": "same",
|
||||
"p_top": 0.999,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"same": 0.999,
|
||||
"elsewhere": 0.001
|
||||
},
|
||||
"blind_top": "same",
|
||||
"input_tokens": 281,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "f-wren-comes",
|
||||
"tag": "case",
|
||||
"decision": "exit2",
|
||||
"label": "meet_wren",
|
||||
"top": "stay",
|
||||
"p_top": 0.996,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"stay": 0.996,
|
||||
"meet_wren": 0.004,
|
||||
"tavern": 0.001
|
||||
},
|
||||
"blind_top": "stay",
|
||||
"input_tokens": 336,
|
||||
"ok": false,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,196 @@
|
||||
{
|
||||
"name": "wyrd-scene-toll",
|
||||
"decisions": [
|
||||
{
|
||||
"id": "place",
|
||||
"question": "This is one turn of an interactive story. Has the scene moved to a genuinely different place during this turn, or does the current scene still hold?",
|
||||
"options": [
|
||||
{
|
||||
"id": "moved",
|
||||
"description": "The scene moved to a genuinely different place."
|
||||
},
|
||||
{
|
||||
"id": "holds",
|
||||
"description": "The current scene still holds."
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "exit",
|
||||
"question": "This is one turn of an interactive story. The story can advance from the current scene into one of the listed next scenes. Which one does this turn move the story into?",
|
||||
"options": [
|
||||
{
|
||||
"id": "stay",
|
||||
"description": "None of these yet. The current scene holds: The First Toll."
|
||||
},
|
||||
{
|
||||
"id": "departure",
|
||||
"description": "Passage Granted/Denied: Beyond the Toll. You leave the tollhouse, either with or without Sera's blessing, continuing down the north road."
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "place2",
|
||||
"question": "This is one turn of an interactive story. The turn began at the north road tollhouse. By the end of this turn's narration, where are the characters?",
|
||||
"options": [
|
||||
{
|
||||
"id": "same",
|
||||
"description": "Still at the north road tollhouse."
|
||||
},
|
||||
{
|
||||
"id": "elsewhere",
|
||||
"description": "Somewhere else: they have left the north road tollhouse for a different place."
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "exit2",
|
||||
"question": "This is one turn of an interactive story. The story can move on from the current scene into one of the listed next scenes, but only once the narration actually gets there. Where does this turn leave the story?",
|
||||
"options": [
|
||||
{
|
||||
"id": "stay",
|
||||
"description": "Still in the current scene, The First Toll: the narration has not reached any next scene yet."
|
||||
},
|
||||
{
|
||||
"id": "departure",
|
||||
"description": "Moved on to Beyond the Toll: You leave the tollhouse, either with or without Sera's blessing, continuing down the north road."
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"cases": [
|
||||
{
|
||||
"id": "t-reads-letter",
|
||||
"tag": "control",
|
||||
"state": {
|
||||
"current_scene": {
|
||||
"title": "The First Toll",
|
||||
"summary": "You arrive at the tollhouse in the pouring rain, presenting your sealed letter to Sera.",
|
||||
"location": "north road tollhouse"
|
||||
},
|
||||
"player_action": "I hand Sera the sealed letter.",
|
||||
"narration": "Sera's gloved hand emerges from the shadow to take the letter, her fingers tracing the wax seal with practiced efficiency. She turns it over slowly, examining it by the light of the single lamp. Rain hammers the iron roof. \"Whose seal is this?\" she asks, without looking up."
|
||||
},
|
||||
"labels": {
|
||||
"place": "holds",
|
||||
"exit": "stay",
|
||||
"place2": "same",
|
||||
"exit2": "stay"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "t-questions",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"current_scene": {
|
||||
"title": "The First Toll",
|
||||
"summary": "You arrive at the tollhouse in the pouring rain, presenting your sealed letter to Sera.",
|
||||
"location": "north road tollhouse"
|
||||
},
|
||||
"player_action": "\"A friend in the capital. That's all you need to know.\"",
|
||||
"narration": "\"That's rarely all I need to know.\" Sera sets the letter on the counter between you and leaves her hand on it. \"The road north has been kind to liars lately. It won't stay that way.\" Water drips from the eaves into a bucket by the door, counting out the silence."
|
||||
},
|
||||
"labels": {
|
||||
"place": "holds",
|
||||
"exit": "stay",
|
||||
"place2": "same",
|
||||
"exit2": "stay"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "t-future",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"current_scene": {
|
||||
"title": "The First Toll",
|
||||
"summary": "You arrive at the tollhouse in the pouring rain, presenting your sealed letter to Sera.",
|
||||
"location": "north road tollhouse"
|
||||
},
|
||||
"player_action": "\"Stamp it and I'll be out of your hair before the rain stops.\"",
|
||||
"narration": "\"Before the rain stops,\" Sera repeats, as if tasting the words. \"It hasn't stopped in nine days.\" She doesn't reach for the stamp. She reaches for the ledger instead, and opens it to a page thick with names, and runs one gloved finger down the column."
|
||||
},
|
||||
"labels": {
|
||||
"place": "holds",
|
||||
"exit": "stay",
|
||||
"place2": "same",
|
||||
"exit2": "stay"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "t-granted",
|
||||
"tag": "control",
|
||||
"state": {
|
||||
"current_scene": {
|
||||
"title": "The First Toll",
|
||||
"summary": "You arrive at the tollhouse in the pouring rain, presenting your sealed letter to Sera.",
|
||||
"location": "north road tollhouse"
|
||||
},
|
||||
"player_action": "I wait while she decides.",
|
||||
"narration": "The stamp comes down with a crack like a gavel. Sera slides the letter back and jerks her chin at the door. You step out into the downpour, and the tollhouse lamp shrinks behind you as the north road unspools ahead, black and glistening, into the hills."
|
||||
},
|
||||
"labels": {
|
||||
"place": "moved",
|
||||
"exit": "departure",
|
||||
"place2": "elsewhere",
|
||||
"exit2": "departure"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "t-denied",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"current_scene": {
|
||||
"title": "The First Toll",
|
||||
"summary": "You arrive at the tollhouse in the pouring rain, presenting your sealed letter to Sera.",
|
||||
"location": "north road tollhouse"
|
||||
},
|
||||
"player_action": "\"Then I'll find another way north.\" I turn to go.",
|
||||
"narration": "\"There is no other way north,\" Sera says to your back, but she lets you go. The door bangs shut behind you. You walk on anyway, past the barrier, the rain soaking through your collar, the tollhouse dwindling to a single yellow window behind you on the north road."
|
||||
},
|
||||
"labels": {
|
||||
"place": "moved",
|
||||
"exit": "departure",
|
||||
"place2": "elsewhere",
|
||||
"exit2": "departure"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "t-rider",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"current_scene": {
|
||||
"title": "The First Toll",
|
||||
"summary": "You arrive at the tollhouse in the pouring rain, presenting your sealed letter to Sera.",
|
||||
"location": "north road tollhouse"
|
||||
},
|
||||
"player_action": "I stay by the stove to dry off.",
|
||||
"narration": "Hooves on the road outside, then a fist on the door. A rider in a sodden Legion cloak ducks in out of the rain, glances at you, glances at Sera, and drops a heavier purse on the counter than any toll requires. Sera's hand closes over it without a word."
|
||||
},
|
||||
"labels": {
|
||||
"place": "holds",
|
||||
"exit": "stay",
|
||||
"place2": "same",
|
||||
"exit2": "stay"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "t-city-memory",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"current_scene": {
|
||||
"title": "The First Toll",
|
||||
"summary": "You arrive at the tollhouse in the pouring rain, presenting your sealed letter to Sera.",
|
||||
"location": "north road tollhouse"
|
||||
},
|
||||
"player_action": "\"You've never been to the capital, have you?\"",
|
||||
"narration": "Something shifts behind Sera's eyes. \"I was born there,\" she says. \"Under the bell tower, in the year the river froze. I left the day they hanged my father in the square.\" She says it flatly, the way you'd read out a toll schedule, and turns back to the ledger."
|
||||
},
|
||||
"labels": {
|
||||
"place": "holds",
|
||||
"exit": "stay",
|
||||
"place2": "same",
|
||||
"exit2": "stay"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,810 @@
|
||||
{
|
||||
"report": {
|
||||
"scenario": "wyrd-scene-toll",
|
||||
"cases": 7,
|
||||
"runs": 3,
|
||||
"latency_per_turn": {
|
||||
"e2e_ms": {
|
||||
"median": 137.7,
|
||||
"min": 136.4,
|
||||
"max": 766.8,
|
||||
"n": 21
|
||||
},
|
||||
"srv_ms": {
|
||||
"median": 106.5,
|
||||
"min": 105.8,
|
||||
"max": 706.7,
|
||||
"n": 21
|
||||
}
|
||||
},
|
||||
"by_decision": {
|
||||
"place": {
|
||||
"evidence": {
|
||||
"n": 7,
|
||||
"correct": 5,
|
||||
"acc": 0.714
|
||||
},
|
||||
"blind_null": {
|
||||
"n": 7,
|
||||
"correct": 5,
|
||||
"acc": 0.714
|
||||
},
|
||||
"controls": {
|
||||
"n": 2,
|
||||
"correct": 1,
|
||||
"acc": 0.5
|
||||
},
|
||||
"non_controls": {
|
||||
"n": 5,
|
||||
"correct": 4,
|
||||
"acc": 0.8
|
||||
},
|
||||
"by_label": {
|
||||
"holds": {
|
||||
"evidence": {
|
||||
"n": 5,
|
||||
"correct": 5,
|
||||
"acc": 1.0
|
||||
},
|
||||
"blind_null": {
|
||||
"n": 5,
|
||||
"correct": 5,
|
||||
"acc": 1.0
|
||||
}
|
||||
},
|
||||
"moved": {
|
||||
"evidence": {
|
||||
"n": 2,
|
||||
"correct": 0,
|
||||
"acc": 0.0
|
||||
},
|
||||
"blind_null": {
|
||||
"n": 2,
|
||||
"correct": 0,
|
||||
"acc": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"unanimous": {
|
||||
"n": 7,
|
||||
"correct": 5,
|
||||
"acc": 0.714
|
||||
},
|
||||
"split": {
|
||||
"n": 0,
|
||||
"correct": 0,
|
||||
"acc": null
|
||||
},
|
||||
"egregious": 0,
|
||||
"blind_egregious": 0,
|
||||
"misses": [
|
||||
"t-granted: said holds (p=0.991, agree=1.0) want moved",
|
||||
"t-denied: said holds (p=0.978, agree=1.0) want moved"
|
||||
]
|
||||
},
|
||||
"exit": {
|
||||
"evidence": {
|
||||
"n": 7,
|
||||
"correct": 5,
|
||||
"acc": 0.714
|
||||
},
|
||||
"blind_null": {
|
||||
"n": 7,
|
||||
"correct": 5,
|
||||
"acc": 0.714
|
||||
},
|
||||
"controls": {
|
||||
"n": 2,
|
||||
"correct": 1,
|
||||
"acc": 0.5
|
||||
},
|
||||
"non_controls": {
|
||||
"n": 5,
|
||||
"correct": 4,
|
||||
"acc": 0.8
|
||||
},
|
||||
"by_label": {
|
||||
"departure": {
|
||||
"evidence": {
|
||||
"n": 2,
|
||||
"correct": 2,
|
||||
"acc": 1.0
|
||||
},
|
||||
"blind_null": {
|
||||
"n": 2,
|
||||
"correct": 0,
|
||||
"acc": 0.0
|
||||
}
|
||||
},
|
||||
"stay": {
|
||||
"evidence": {
|
||||
"n": 5,
|
||||
"correct": 3,
|
||||
"acc": 0.6
|
||||
},
|
||||
"blind_null": {
|
||||
"n": 5,
|
||||
"correct": 5,
|
||||
"acc": 1.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"unanimous": {
|
||||
"n": 4,
|
||||
"correct": 3,
|
||||
"acc": 0.75
|
||||
},
|
||||
"split": {
|
||||
"n": 3,
|
||||
"correct": 2,
|
||||
"acc": 0.667
|
||||
},
|
||||
"egregious": 0,
|
||||
"blind_egregious": 0,
|
||||
"misses": [
|
||||
"t-reads-letter: said departure (p=0.593, agree=0.5) want stay",
|
||||
"t-questions: said departure (p=0.665, agree=1.0) want stay"
|
||||
]
|
||||
},
|
||||
"place2": {
|
||||
"evidence": {
|
||||
"n": 7,
|
||||
"correct": 7,
|
||||
"acc": 1.0
|
||||
},
|
||||
"blind_null": {
|
||||
"n": 7,
|
||||
"correct": 5,
|
||||
"acc": 0.714
|
||||
},
|
||||
"controls": {
|
||||
"n": 2,
|
||||
"correct": 2,
|
||||
"acc": 1.0
|
||||
},
|
||||
"non_controls": {
|
||||
"n": 5,
|
||||
"correct": 5,
|
||||
"acc": 1.0
|
||||
},
|
||||
"by_label": {
|
||||
"elsewhere": {
|
||||
"evidence": {
|
||||
"n": 2,
|
||||
"correct": 2,
|
||||
"acc": 1.0
|
||||
},
|
||||
"blind_null": {
|
||||
"n": 2,
|
||||
"correct": 0,
|
||||
"acc": 0.0
|
||||
}
|
||||
},
|
||||
"same": {
|
||||
"evidence": {
|
||||
"n": 5,
|
||||
"correct": 5,
|
||||
"acc": 1.0
|
||||
},
|
||||
"blind_null": {
|
||||
"n": 5,
|
||||
"correct": 5,
|
||||
"acc": 1.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"unanimous": {
|
||||
"n": 7,
|
||||
"correct": 7,
|
||||
"acc": 1.0
|
||||
},
|
||||
"split": {
|
||||
"n": 0,
|
||||
"correct": 0,
|
||||
"acc": null
|
||||
},
|
||||
"egregious": 0,
|
||||
"blind_egregious": 0,
|
||||
"misses": []
|
||||
},
|
||||
"exit2": {
|
||||
"evidence": {
|
||||
"n": 7,
|
||||
"correct": 5,
|
||||
"acc": 0.714
|
||||
},
|
||||
"blind_null": {
|
||||
"n": 7,
|
||||
"correct": 5,
|
||||
"acc": 0.714
|
||||
},
|
||||
"controls": {
|
||||
"n": 2,
|
||||
"correct": 1,
|
||||
"acc": 0.5
|
||||
},
|
||||
"non_controls": {
|
||||
"n": 5,
|
||||
"correct": 4,
|
||||
"acc": 0.8
|
||||
},
|
||||
"by_label": {
|
||||
"departure": {
|
||||
"evidence": {
|
||||
"n": 2,
|
||||
"correct": 0,
|
||||
"acc": 0.0
|
||||
},
|
||||
"blind_null": {
|
||||
"n": 2,
|
||||
"correct": 0,
|
||||
"acc": 0.0
|
||||
}
|
||||
},
|
||||
"stay": {
|
||||
"evidence": {
|
||||
"n": 5,
|
||||
"correct": 5,
|
||||
"acc": 1.0
|
||||
},
|
||||
"blind_null": {
|
||||
"n": 5,
|
||||
"correct": 5,
|
||||
"acc": 1.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"unanimous": {
|
||||
"n": 6,
|
||||
"correct": 5,
|
||||
"acc": 0.833
|
||||
},
|
||||
"split": {
|
||||
"n": 1,
|
||||
"correct": 0,
|
||||
"acc": 0.0
|
||||
},
|
||||
"egregious": 0,
|
||||
"blind_egregious": 0,
|
||||
"misses": [
|
||||
"t-granted: said stay (p=0.978, agree=1.0) want departure",
|
||||
"t-denied: said stay (p=0.637, agree=0.5) want departure"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"rows": [
|
||||
{
|
||||
"case": "t-reads-letter",
|
||||
"tag": "control",
|
||||
"decision": "place",
|
||||
"label": "holds",
|
||||
"top": "holds",
|
||||
"p_top": 1.0,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"moved": 0.0,
|
||||
"holds": 1.0
|
||||
},
|
||||
"blind_top": "holds",
|
||||
"input_tokens": 250,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "t-reads-letter",
|
||||
"tag": "control",
|
||||
"decision": "exit",
|
||||
"label": "stay",
|
||||
"top": "departure",
|
||||
"p_top": 0.593,
|
||||
"agreement": 0.5,
|
||||
"probs": {
|
||||
"stay": 0.407,
|
||||
"departure": 0.593
|
||||
},
|
||||
"blind_top": "stay",
|
||||
"input_tokens": 285,
|
||||
"ok": false,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "t-reads-letter",
|
||||
"tag": "control",
|
||||
"decision": "place2",
|
||||
"label": "same",
|
||||
"top": "same",
|
||||
"p_top": 1.0,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"same": 1.0,
|
||||
"elsewhere": 0.0
|
||||
},
|
||||
"blind_top": "same",
|
||||
"input_tokens": 263,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "t-reads-letter",
|
||||
"tag": "control",
|
||||
"decision": "exit2",
|
||||
"label": "stay",
|
||||
"top": "stay",
|
||||
"p_top": 1.0,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"stay": 1.0,
|
||||
"departure": 0.0
|
||||
},
|
||||
"blind_top": "stay",
|
||||
"input_tokens": 295,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "t-questions",
|
||||
"tag": "case",
|
||||
"decision": "place",
|
||||
"label": "holds",
|
||||
"top": "holds",
|
||||
"p_top": 0.999,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"moved": 0.001,
|
||||
"holds": 0.999
|
||||
},
|
||||
"blind_top": "holds",
|
||||
"input_tokens": 258,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "t-questions",
|
||||
"tag": "case",
|
||||
"decision": "exit",
|
||||
"label": "stay",
|
||||
"top": "departure",
|
||||
"p_top": 0.665,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"stay": 0.335,
|
||||
"departure": 0.665
|
||||
},
|
||||
"blind_top": "stay",
|
||||
"input_tokens": 293,
|
||||
"ok": false,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "t-questions",
|
||||
"tag": "case",
|
||||
"decision": "place2",
|
||||
"label": "same",
|
||||
"top": "same",
|
||||
"p_top": 0.999,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"same": 0.999,
|
||||
"elsewhere": 0.001
|
||||
},
|
||||
"blind_top": "same",
|
||||
"input_tokens": 271,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "t-questions",
|
||||
"tag": "case",
|
||||
"decision": "exit2",
|
||||
"label": "stay",
|
||||
"top": "stay",
|
||||
"p_top": 0.999,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"stay": 0.999,
|
||||
"departure": 0.001
|
||||
},
|
||||
"blind_top": "stay",
|
||||
"input_tokens": 303,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "t-future",
|
||||
"tag": "case",
|
||||
"decision": "place",
|
||||
"label": "holds",
|
||||
"top": "holds",
|
||||
"p_top": 0.999,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"moved": 0.001,
|
||||
"holds": 0.999
|
||||
},
|
||||
"blind_top": "holds",
|
||||
"input_tokens": 254,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "t-future",
|
||||
"tag": "case",
|
||||
"decision": "exit",
|
||||
"label": "stay",
|
||||
"top": "stay",
|
||||
"p_top": 0.693,
|
||||
"agreement": 0.5,
|
||||
"probs": {
|
||||
"stay": 0.693,
|
||||
"departure": 0.307
|
||||
},
|
||||
"blind_top": "stay",
|
||||
"input_tokens": 289,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "t-future",
|
||||
"tag": "case",
|
||||
"decision": "place2",
|
||||
"label": "same",
|
||||
"top": "same",
|
||||
"p_top": 0.999,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"same": 0.999,
|
||||
"elsewhere": 0.001
|
||||
},
|
||||
"blind_top": "same",
|
||||
"input_tokens": 267,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "t-future",
|
||||
"tag": "case",
|
||||
"decision": "exit2",
|
||||
"label": "stay",
|
||||
"top": "stay",
|
||||
"p_top": 0.999,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"stay": 0.999,
|
||||
"departure": 0.001
|
||||
},
|
||||
"blind_top": "stay",
|
||||
"input_tokens": 299,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "t-granted",
|
||||
"tag": "control",
|
||||
"decision": "place",
|
||||
"label": "moved",
|
||||
"top": "holds",
|
||||
"p_top": 0.991,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"moved": 0.009,
|
||||
"holds": 0.991
|
||||
},
|
||||
"blind_top": "holds",
|
||||
"input_tokens": 247,
|
||||
"ok": false,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "t-granted",
|
||||
"tag": "control",
|
||||
"decision": "exit",
|
||||
"label": "departure",
|
||||
"top": "departure",
|
||||
"p_top": 0.852,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"stay": 0.148,
|
||||
"departure": 0.852
|
||||
},
|
||||
"blind_top": "stay",
|
||||
"input_tokens": 282,
|
||||
"ok": true,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "t-granted",
|
||||
"tag": "control",
|
||||
"decision": "place2",
|
||||
"label": "elsewhere",
|
||||
"top": "elsewhere",
|
||||
"p_top": 0.96,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"same": 0.04,
|
||||
"elsewhere": 0.96
|
||||
},
|
||||
"blind_top": "same",
|
||||
"input_tokens": 260,
|
||||
"ok": true,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "t-granted",
|
||||
"tag": "control",
|
||||
"decision": "exit2",
|
||||
"label": "departure",
|
||||
"top": "stay",
|
||||
"p_top": 0.978,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"stay": 0.978,
|
||||
"departure": 0.022
|
||||
},
|
||||
"blind_top": "stay",
|
||||
"input_tokens": 292,
|
||||
"ok": false,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "t-denied",
|
||||
"tag": "case",
|
||||
"decision": "place",
|
||||
"label": "moved",
|
||||
"top": "holds",
|
||||
"p_top": 0.978,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"moved": 0.022,
|
||||
"holds": 0.978
|
||||
},
|
||||
"blind_top": "holds",
|
||||
"input_tokens": 253,
|
||||
"ok": false,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "t-denied",
|
||||
"tag": "case",
|
||||
"decision": "exit",
|
||||
"label": "departure",
|
||||
"top": "departure",
|
||||
"p_top": 0.95,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"stay": 0.05,
|
||||
"departure": 0.95
|
||||
},
|
||||
"blind_top": "stay",
|
||||
"input_tokens": 288,
|
||||
"ok": true,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "t-denied",
|
||||
"tag": "case",
|
||||
"decision": "place2",
|
||||
"label": "elsewhere",
|
||||
"top": "elsewhere",
|
||||
"p_top": 0.91,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"same": 0.09,
|
||||
"elsewhere": 0.91
|
||||
},
|
||||
"blind_top": "same",
|
||||
"input_tokens": 266,
|
||||
"ok": true,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "t-denied",
|
||||
"tag": "case",
|
||||
"decision": "exit2",
|
||||
"label": "departure",
|
||||
"top": "stay",
|
||||
"p_top": 0.637,
|
||||
"agreement": 0.5,
|
||||
"probs": {
|
||||
"stay": 0.637,
|
||||
"departure": 0.363
|
||||
},
|
||||
"blind_top": "stay",
|
||||
"input_tokens": 298,
|
||||
"ok": false,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "t-rider",
|
||||
"tag": "case",
|
||||
"decision": "place",
|
||||
"label": "holds",
|
||||
"top": "holds",
|
||||
"p_top": 0.998,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"moved": 0.002,
|
||||
"holds": 0.998
|
||||
},
|
||||
"blind_top": "holds",
|
||||
"input_tokens": 251,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "t-rider",
|
||||
"tag": "case",
|
||||
"decision": "exit",
|
||||
"label": "stay",
|
||||
"top": "stay",
|
||||
"p_top": 0.622,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"stay": 0.622,
|
||||
"departure": 0.378
|
||||
},
|
||||
"blind_top": "stay",
|
||||
"input_tokens": 286,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "t-rider",
|
||||
"tag": "case",
|
||||
"decision": "place2",
|
||||
"label": "same",
|
||||
"top": "same",
|
||||
"p_top": 1.0,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"same": 1.0,
|
||||
"elsewhere": 0.0
|
||||
},
|
||||
"blind_top": "same",
|
||||
"input_tokens": 264,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "t-rider",
|
||||
"tag": "case",
|
||||
"decision": "exit2",
|
||||
"label": "stay",
|
||||
"top": "stay",
|
||||
"p_top": 0.999,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"stay": 0.999,
|
||||
"departure": 0.001
|
||||
},
|
||||
"blind_top": "stay",
|
||||
"input_tokens": 296,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "t-city-memory",
|
||||
"tag": "case",
|
||||
"decision": "place",
|
||||
"label": "holds",
|
||||
"top": "holds",
|
||||
"p_top": 0.999,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"moved": 0.001,
|
||||
"holds": 0.999
|
||||
},
|
||||
"blind_top": "holds",
|
||||
"input_tokens": 257,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "t-city-memory",
|
||||
"tag": "case",
|
||||
"decision": "exit",
|
||||
"label": "stay",
|
||||
"top": "stay",
|
||||
"p_top": 0.593,
|
||||
"agreement": 0.5,
|
||||
"probs": {
|
||||
"stay": 0.593,
|
||||
"departure": 0.407
|
||||
},
|
||||
"blind_top": "stay",
|
||||
"input_tokens": 292,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "t-city-memory",
|
||||
"tag": "case",
|
||||
"decision": "place2",
|
||||
"label": "same",
|
||||
"top": "same",
|
||||
"p_top": 0.999,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"same": 0.999,
|
||||
"elsewhere": 0.001
|
||||
},
|
||||
"blind_top": "same",
|
||||
"input_tokens": 270,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "t-city-memory",
|
||||
"tag": "case",
|
||||
"decision": "exit2",
|
||||
"label": "stay",
|
||||
"top": "stay",
|
||||
"p_top": 0.999,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"stay": 0.999,
|
||||
"departure": 0.001
|
||||
},
|
||||
"blind_top": "stay",
|
||||
"input_tokens": 302,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,177 @@
|
||||
{
|
||||
"name": "wyrd-scene-vigil",
|
||||
"decisions": [
|
||||
{
|
||||
"id": "place",
|
||||
"question": "This is one turn of an interactive story. Has the scene moved to a genuinely different place during this turn, or does the current scene still hold?",
|
||||
"options": [
|
||||
{
|
||||
"id": "moved",
|
||||
"description": "The scene moved to a genuinely different place."
|
||||
},
|
||||
{
|
||||
"id": "holds",
|
||||
"description": "The current scene still holds."
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "exit",
|
||||
"question": "This is one turn of an interactive story. The story can advance from the current scene into one of the listed next scenes. Which one does this turn move the story into?",
|
||||
"options": [
|
||||
{
|
||||
"id": "stay",
|
||||
"description": "None of these yet. The current scene holds: The Last Vigil."
|
||||
},
|
||||
{
|
||||
"id": "outside",
|
||||
"description": "Leave the sanctum: Outside the Sanctum. Lynnae and the player step out of the interior into the exterior surroundings."
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "place2",
|
||||
"question": "This is one turn of an interactive story. The turn began at the sanctum interior. By the end of this turn's narration, where are the characters?",
|
||||
"options": [
|
||||
{
|
||||
"id": "same",
|
||||
"description": "Still at the sanctum interior."
|
||||
},
|
||||
{
|
||||
"id": "elsewhere",
|
||||
"description": "Somewhere else: they have left the sanctum interior for a different place."
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "exit2",
|
||||
"question": "This is one turn of an interactive story. The story can move on from the current scene into one of the listed next scenes, but only once the narration actually gets there. Where does this turn leave the story?",
|
||||
"options": [
|
||||
{
|
||||
"id": "stay",
|
||||
"description": "Still in the current scene, The Last Vigil: the narration has not reached any next scene yet."
|
||||
},
|
||||
{
|
||||
"id": "outside",
|
||||
"description": "Moved on to Outside the Sanctum: Lynnae and the player step out of the interior into the exterior surroundings."
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"cases": [
|
||||
{
|
||||
"id": "v-talk",
|
||||
"tag": "control",
|
||||
"state": {
|
||||
"current_scene": {
|
||||
"title": "The Last Vigil",
|
||||
"summary": "The player stands in the inner sanctum, guarding the Book. Lynnae enters through a hidden alcove to discuss the encroaching demons.",
|
||||
"location": "sanctum interior"
|
||||
},
|
||||
"player_action": "\"How close are they?\"",
|
||||
"narration": "Lynnae rests her fingertips on the edge of the altar, not quite touching the Book. \"Close enough that the wards are singing,\" she says. \"Listen.\" You do, and beneath the wind there is a high, thin note, like a wet finger on the rim of a glass. The candles lean toward the door."
|
||||
},
|
||||
"labels": {
|
||||
"place": "holds",
|
||||
"exit": "stay",
|
||||
"place2": "same",
|
||||
"exit2": "stay"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "v-step-out",
|
||||
"tag": "control",
|
||||
"state": {
|
||||
"current_scene": {
|
||||
"title": "The Last Vigil",
|
||||
"summary": "The player stands in the inner sanctum, guarding the Book. Lynnae enters through a hidden alcove to discuss the encroaching demons.",
|
||||
"location": "sanctum interior"
|
||||
},
|
||||
"player_action": "\"Show me.\" I follow her out through the great doors.",
|
||||
"narration": "The doors groan open and the wind takes the candle flames behind you. Outside, the sanctum steps drop away into a valley of ash, and on the far ridge a line of red lights moves slowly, patiently, like a procession of lanterns that has forgotten what it is mourning."
|
||||
},
|
||||
"labels": {
|
||||
"place": "moved",
|
||||
"exit": "outside",
|
||||
"place2": "elsewhere",
|
||||
"exit2": "outside"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "v-plan-pub",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"current_scene": {
|
||||
"title": "The Last Vigil",
|
||||
"summary": "The player stands in the inner sanctum, guarding the Book. Lynnae enters through a hidden alcove to discuss the encroaching demons.",
|
||||
"location": "sanctum interior"
|
||||
},
|
||||
"player_action": "\"If the wards fall, we take the Book to the old pub in the village.\"",
|
||||
"narration": "\"The Drowned Crown?\" Lynnae almost smiles. \"No demon would look for holy relics behind a bar.\" She turns the idea over, then nods slowly. \"If it comes to that. But the wards have not fallen yet, and I would rather they never did.\""
|
||||
},
|
||||
"labels": {
|
||||
"place": "holds",
|
||||
"exit": "stay",
|
||||
"place2": "same",
|
||||
"exit2": "stay"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "v-to-pub",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"current_scene": {
|
||||
"title": "The Last Vigil",
|
||||
"summary": "The player stands in the inner sanctum, guarding the Book. Lynnae enters through a hidden alcove to discuss the encroaching demons.",
|
||||
"location": "sanctum interior"
|
||||
},
|
||||
"player_action": "I take the Book and we run for the village pub.",
|
||||
"narration": "You run through the ash with the Book clutched to your chest, Lynnae a step behind. The heavy door creaks on rusted hinges as you cross the threshold, the air inside the tavern thick with the scent of stale ale, damp rot, and cold ash. You set the Book down on the bar."
|
||||
},
|
||||
"labels": {
|
||||
"place": "moved",
|
||||
"exit": "stay",
|
||||
"place2": "elsewhere",
|
||||
"exit2": "stay"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "v-vision",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"current_scene": {
|
||||
"title": "The Last Vigil",
|
||||
"summary": "The player stands in the inner sanctum, guarding the Book. Lynnae enters through a hidden alcove to discuss the encroaching demons.",
|
||||
"location": "sanctum interior"
|
||||
},
|
||||
"player_action": "I touch the Book's cover.",
|
||||
"narration": "The world tilts. For one breath you are elsewhere: a burning library, a hundred years ago, monks carrying armfuls of scrolls through smoke. Then the vision snaps shut like a book, and you are back in the sanctum with Lynnae's hand hard on your wrist. \"Don't,\" she says. \"Not yet.\""
|
||||
},
|
||||
"labels": {
|
||||
"place": "holds",
|
||||
"exit": "stay",
|
||||
"place2": "same",
|
||||
"exit2": "stay"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "v-demon-enters",
|
||||
"tag": "case",
|
||||
"state": {
|
||||
"current_scene": {
|
||||
"title": "The Last Vigil",
|
||||
"summary": "The player stands in the inner sanctum, guarding the Book. Lynnae enters through a hidden alcove to discuss the encroaching demons.",
|
||||
"location": "sanctum interior"
|
||||
},
|
||||
"player_action": "I draw my blade and stand before the altar.",
|
||||
"narration": "The great doors burst inward. Ash rolls across the floor, and in the middle of it something tall unfolds itself, too many joints, a crown of red light. The candles gutter out all at once. Lynnae is already chanting, her voice steady, her hands trembling."
|
||||
},
|
||||
"labels": {
|
||||
"place": "holds",
|
||||
"exit": "stay",
|
||||
"place2": "same",
|
||||
"exit2": "stay"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,730 @@
|
||||
{
|
||||
"report": {
|
||||
"scenario": "wyrd-scene-vigil",
|
||||
"cases": 6,
|
||||
"runs": 3,
|
||||
"latency_per_turn": {
|
||||
"e2e_ms": {
|
||||
"median": 133.2,
|
||||
"min": 130.8,
|
||||
"max": 155.9,
|
||||
"n": 18
|
||||
},
|
||||
"srv_ms": {
|
||||
"median": 101.1,
|
||||
"min": 100.4,
|
||||
"max": 101.8,
|
||||
"n": 18
|
||||
}
|
||||
},
|
||||
"by_decision": {
|
||||
"place": {
|
||||
"evidence": {
|
||||
"n": 6,
|
||||
"correct": 4,
|
||||
"acc": 0.667
|
||||
},
|
||||
"blind_null": {
|
||||
"n": 6,
|
||||
"correct": 4,
|
||||
"acc": 0.667
|
||||
},
|
||||
"controls": {
|
||||
"n": 2,
|
||||
"correct": 1,
|
||||
"acc": 0.5
|
||||
},
|
||||
"non_controls": {
|
||||
"n": 4,
|
||||
"correct": 3,
|
||||
"acc": 0.75
|
||||
},
|
||||
"by_label": {
|
||||
"holds": {
|
||||
"evidence": {
|
||||
"n": 4,
|
||||
"correct": 4,
|
||||
"acc": 1.0
|
||||
},
|
||||
"blind_null": {
|
||||
"n": 4,
|
||||
"correct": 4,
|
||||
"acc": 1.0
|
||||
}
|
||||
},
|
||||
"moved": {
|
||||
"evidence": {
|
||||
"n": 2,
|
||||
"correct": 0,
|
||||
"acc": 0.0
|
||||
},
|
||||
"blind_null": {
|
||||
"n": 2,
|
||||
"correct": 0,
|
||||
"acc": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"unanimous": {
|
||||
"n": 5,
|
||||
"correct": 4,
|
||||
"acc": 0.8
|
||||
},
|
||||
"split": {
|
||||
"n": 1,
|
||||
"correct": 0,
|
||||
"acc": 0.0
|
||||
},
|
||||
"egregious": 0,
|
||||
"blind_egregious": 0,
|
||||
"misses": [
|
||||
"v-step-out: said holds (p=0.731, agree=1.0) want moved",
|
||||
"v-to-pub: said holds (p=0.637, agree=0.5) want moved"
|
||||
]
|
||||
},
|
||||
"exit": {
|
||||
"evidence": {
|
||||
"n": 6,
|
||||
"correct": 5,
|
||||
"acc": 0.833
|
||||
},
|
||||
"blind_null": {
|
||||
"n": 6,
|
||||
"correct": 5,
|
||||
"acc": 0.833
|
||||
},
|
||||
"controls": {
|
||||
"n": 2,
|
||||
"correct": 2,
|
||||
"acc": 1.0
|
||||
},
|
||||
"non_controls": {
|
||||
"n": 4,
|
||||
"correct": 3,
|
||||
"acc": 0.75
|
||||
},
|
||||
"by_label": {
|
||||
"outside": {
|
||||
"evidence": {
|
||||
"n": 1,
|
||||
"correct": 1,
|
||||
"acc": 1.0
|
||||
},
|
||||
"blind_null": {
|
||||
"n": 1,
|
||||
"correct": 0,
|
||||
"acc": 0.0
|
||||
}
|
||||
},
|
||||
"stay": {
|
||||
"evidence": {
|
||||
"n": 5,
|
||||
"correct": 4,
|
||||
"acc": 0.8
|
||||
},
|
||||
"blind_null": {
|
||||
"n": 5,
|
||||
"correct": 5,
|
||||
"acc": 1.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"unanimous": {
|
||||
"n": 6,
|
||||
"correct": 5,
|
||||
"acc": 0.833
|
||||
},
|
||||
"split": {
|
||||
"n": 0,
|
||||
"correct": 0,
|
||||
"acc": null
|
||||
},
|
||||
"egregious": 0,
|
||||
"blind_egregious": 0,
|
||||
"misses": [
|
||||
"v-to-pub: said outside (p=0.86, agree=1.0) want stay"
|
||||
]
|
||||
},
|
||||
"place2": {
|
||||
"evidence": {
|
||||
"n": 6,
|
||||
"correct": 6,
|
||||
"acc": 1.0
|
||||
},
|
||||
"blind_null": {
|
||||
"n": 6,
|
||||
"correct": 4,
|
||||
"acc": 0.667
|
||||
},
|
||||
"controls": {
|
||||
"n": 2,
|
||||
"correct": 2,
|
||||
"acc": 1.0
|
||||
},
|
||||
"non_controls": {
|
||||
"n": 4,
|
||||
"correct": 4,
|
||||
"acc": 1.0
|
||||
},
|
||||
"by_label": {
|
||||
"elsewhere": {
|
||||
"evidence": {
|
||||
"n": 2,
|
||||
"correct": 2,
|
||||
"acc": 1.0
|
||||
},
|
||||
"blind_null": {
|
||||
"n": 2,
|
||||
"correct": 0,
|
||||
"acc": 0.0
|
||||
}
|
||||
},
|
||||
"same": {
|
||||
"evidence": {
|
||||
"n": 4,
|
||||
"correct": 4,
|
||||
"acc": 1.0
|
||||
},
|
||||
"blind_null": {
|
||||
"n": 4,
|
||||
"correct": 4,
|
||||
"acc": 1.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"unanimous": {
|
||||
"n": 6,
|
||||
"correct": 6,
|
||||
"acc": 1.0
|
||||
},
|
||||
"split": {
|
||||
"n": 0,
|
||||
"correct": 0,
|
||||
"acc": null
|
||||
},
|
||||
"egregious": 0,
|
||||
"blind_egregious": 0,
|
||||
"misses": []
|
||||
},
|
||||
"exit2": {
|
||||
"evidence": {
|
||||
"n": 6,
|
||||
"correct": 6,
|
||||
"acc": 1.0
|
||||
},
|
||||
"blind_null": {
|
||||
"n": 6,
|
||||
"correct": 5,
|
||||
"acc": 0.833
|
||||
},
|
||||
"controls": {
|
||||
"n": 2,
|
||||
"correct": 2,
|
||||
"acc": 1.0
|
||||
},
|
||||
"non_controls": {
|
||||
"n": 4,
|
||||
"correct": 4,
|
||||
"acc": 1.0
|
||||
},
|
||||
"by_label": {
|
||||
"outside": {
|
||||
"evidence": {
|
||||
"n": 1,
|
||||
"correct": 1,
|
||||
"acc": 1.0
|
||||
},
|
||||
"blind_null": {
|
||||
"n": 1,
|
||||
"correct": 0,
|
||||
"acc": 0.0
|
||||
}
|
||||
},
|
||||
"stay": {
|
||||
"evidence": {
|
||||
"n": 5,
|
||||
"correct": 5,
|
||||
"acc": 1.0
|
||||
},
|
||||
"blind_null": {
|
||||
"n": 5,
|
||||
"correct": 5,
|
||||
"acc": 1.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"unanimous": {
|
||||
"n": 6,
|
||||
"correct": 6,
|
||||
"acc": 1.0
|
||||
},
|
||||
"split": {
|
||||
"n": 0,
|
||||
"correct": 0,
|
||||
"acc": null
|
||||
},
|
||||
"egregious": 0,
|
||||
"blind_egregious": 0,
|
||||
"misses": []
|
||||
}
|
||||
}
|
||||
},
|
||||
"rows": [
|
||||
{
|
||||
"case": "v-talk",
|
||||
"tag": "control",
|
||||
"decision": "place",
|
||||
"label": "holds",
|
||||
"top": "holds",
|
||||
"p_top": 0.999,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"moved": 0.001,
|
||||
"holds": 0.999
|
||||
},
|
||||
"blind_top": "holds",
|
||||
"input_tokens": 266,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "v-talk",
|
||||
"tag": "control",
|
||||
"decision": "exit",
|
||||
"label": "stay",
|
||||
"top": "stay",
|
||||
"p_top": 0.86,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"stay": 0.86,
|
||||
"outside": 0.14
|
||||
},
|
||||
"blind_top": "stay",
|
||||
"input_tokens": 296,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "v-talk",
|
||||
"tag": "control",
|
||||
"decision": "place2",
|
||||
"label": "same",
|
||||
"top": "same",
|
||||
"p_top": 0.999,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"same": 0.999,
|
||||
"elsewhere": 0.001
|
||||
},
|
||||
"blind_top": "same",
|
||||
"input_tokens": 276,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "v-talk",
|
||||
"tag": "control",
|
||||
"decision": "exit2",
|
||||
"label": "stay",
|
||||
"top": "stay",
|
||||
"p_top": 1.0,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"stay": 1.0,
|
||||
"outside": 0.0
|
||||
},
|
||||
"blind_top": "stay",
|
||||
"input_tokens": 308,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "v-step-out",
|
||||
"tag": "control",
|
||||
"decision": "place",
|
||||
"label": "moved",
|
||||
"top": "holds",
|
||||
"p_top": 0.731,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"moved": 0.269,
|
||||
"holds": 0.731
|
||||
},
|
||||
"blind_top": "holds",
|
||||
"input_tokens": 262,
|
||||
"ok": false,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "v-step-out",
|
||||
"tag": "control",
|
||||
"decision": "exit",
|
||||
"label": "outside",
|
||||
"top": "outside",
|
||||
"p_top": 0.998,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"stay": 0.002,
|
||||
"outside": 0.998
|
||||
},
|
||||
"blind_top": "stay",
|
||||
"input_tokens": 292,
|
||||
"ok": true,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "v-step-out",
|
||||
"tag": "control",
|
||||
"decision": "place2",
|
||||
"label": "elsewhere",
|
||||
"top": "elsewhere",
|
||||
"p_top": 0.997,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"same": 0.003,
|
||||
"elsewhere": 0.997
|
||||
},
|
||||
"blind_top": "same",
|
||||
"input_tokens": 272,
|
||||
"ok": true,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "v-step-out",
|
||||
"tag": "control",
|
||||
"decision": "exit2",
|
||||
"label": "outside",
|
||||
"top": "outside",
|
||||
"p_top": 0.835,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"stay": 0.165,
|
||||
"outside": 0.835
|
||||
},
|
||||
"blind_top": "stay",
|
||||
"input_tokens": 304,
|
||||
"ok": true,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "v-plan-pub",
|
||||
"tag": "case",
|
||||
"decision": "place",
|
||||
"label": "holds",
|
||||
"top": "holds",
|
||||
"p_top": 0.999,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"moved": 0.001,
|
||||
"holds": 0.999
|
||||
},
|
||||
"blind_top": "holds",
|
||||
"input_tokens": 267,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "v-plan-pub",
|
||||
"tag": "case",
|
||||
"decision": "exit",
|
||||
"label": "stay",
|
||||
"top": "stay",
|
||||
"p_top": 0.844,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"stay": 0.844,
|
||||
"outside": 0.156
|
||||
},
|
||||
"blind_top": "stay",
|
||||
"input_tokens": 297,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "v-plan-pub",
|
||||
"tag": "case",
|
||||
"decision": "place2",
|
||||
"label": "same",
|
||||
"top": "same",
|
||||
"p_top": 0.986,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"same": 0.986,
|
||||
"elsewhere": 0.014
|
||||
},
|
||||
"blind_top": "same",
|
||||
"input_tokens": 277,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "v-plan-pub",
|
||||
"tag": "case",
|
||||
"decision": "exit2",
|
||||
"label": "stay",
|
||||
"top": "stay",
|
||||
"p_top": 1.0,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"stay": 1.0,
|
||||
"outside": 0.0
|
||||
},
|
||||
"blind_top": "stay",
|
||||
"input_tokens": 309,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "v-to-pub",
|
||||
"tag": "case",
|
||||
"decision": "place",
|
||||
"label": "moved",
|
||||
"top": "holds",
|
||||
"p_top": 0.637,
|
||||
"agreement": 0.5,
|
||||
"probs": {
|
||||
"moved": 0.363,
|
||||
"holds": 0.637
|
||||
},
|
||||
"blind_top": "holds",
|
||||
"input_tokens": 271,
|
||||
"ok": false,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "v-to-pub",
|
||||
"tag": "case",
|
||||
"decision": "exit",
|
||||
"label": "stay",
|
||||
"top": "outside",
|
||||
"p_top": 0.86,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"stay": 0.14,
|
||||
"outside": 0.86
|
||||
},
|
||||
"blind_top": "stay",
|
||||
"input_tokens": 301,
|
||||
"ok": false,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "v-to-pub",
|
||||
"tag": "case",
|
||||
"decision": "place2",
|
||||
"label": "elsewhere",
|
||||
"top": "elsewhere",
|
||||
"p_top": 0.999,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"same": 0.001,
|
||||
"elsewhere": 0.999
|
||||
},
|
||||
"blind_top": "same",
|
||||
"input_tokens": 281,
|
||||
"ok": true,
|
||||
"blind_ok": false,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "v-to-pub",
|
||||
"tag": "case",
|
||||
"decision": "exit2",
|
||||
"label": "stay",
|
||||
"top": "stay",
|
||||
"p_top": 0.943,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"stay": 0.943,
|
||||
"outside": 0.057
|
||||
},
|
||||
"blind_top": "stay",
|
||||
"input_tokens": 313,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "v-vision",
|
||||
"tag": "case",
|
||||
"decision": "place",
|
||||
"label": "holds",
|
||||
"top": "holds",
|
||||
"p_top": 0.999,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"moved": 0.001,
|
||||
"holds": 0.999
|
||||
},
|
||||
"blind_top": "holds",
|
||||
"input_tokens": 269,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "v-vision",
|
||||
"tag": "case",
|
||||
"decision": "exit",
|
||||
"label": "stay",
|
||||
"top": "stay",
|
||||
"p_top": 0.881,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"stay": 0.881,
|
||||
"outside": 0.119
|
||||
},
|
||||
"blind_top": "stay",
|
||||
"input_tokens": 299,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "v-vision",
|
||||
"tag": "case",
|
||||
"decision": "place2",
|
||||
"label": "same",
|
||||
"top": "same",
|
||||
"p_top": 0.991,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"same": 0.991,
|
||||
"elsewhere": 0.009
|
||||
},
|
||||
"blind_top": "same",
|
||||
"input_tokens": 279,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "v-vision",
|
||||
"tag": "case",
|
||||
"decision": "exit2",
|
||||
"label": "stay",
|
||||
"top": "stay",
|
||||
"p_top": 1.0,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"stay": 1.0,
|
||||
"outside": 0.0
|
||||
},
|
||||
"blind_top": "stay",
|
||||
"input_tokens": 311,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "v-demon-enters",
|
||||
"tag": "case",
|
||||
"decision": "place",
|
||||
"label": "holds",
|
||||
"top": "holds",
|
||||
"p_top": 0.994,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"moved": 0.006,
|
||||
"holds": 0.994
|
||||
},
|
||||
"blind_top": "holds",
|
||||
"input_tokens": 258,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "v-demon-enters",
|
||||
"tag": "case",
|
||||
"decision": "exit",
|
||||
"label": "stay",
|
||||
"top": "stay",
|
||||
"p_top": 0.86,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"stay": 0.86,
|
||||
"outside": 0.14
|
||||
},
|
||||
"blind_top": "stay",
|
||||
"input_tokens": 288,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "v-demon-enters",
|
||||
"tag": "case",
|
||||
"decision": "place2",
|
||||
"label": "same",
|
||||
"top": "same",
|
||||
"p_top": 0.993,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"same": 0.993,
|
||||
"elsewhere": 0.007
|
||||
},
|
||||
"blind_top": "same",
|
||||
"input_tokens": 268,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
},
|
||||
{
|
||||
"case": "v-demon-enters",
|
||||
"tag": "case",
|
||||
"decision": "exit2",
|
||||
"label": "stay",
|
||||
"top": "stay",
|
||||
"p_top": 0.999,
|
||||
"agreement": 1.0,
|
||||
"probs": {
|
||||
"stay": 0.999,
|
||||
"outside": 0.001
|
||||
},
|
||||
"blind_top": "stay",
|
||||
"input_tokens": 300,
|
||||
"ok": true,
|
||||
"blind_ok": true,
|
||||
"egregious": false,
|
||||
"blind_egregious": false
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,104 @@
|
||||
"""SPIKE (2026-09-27): is semif-serve a clean fit for a consumer's per-turn decision?
|
||||
Throwaway measurement, no service change. A scenario file names the decisions a consumer makes
|
||||
per turn and a set of hand-labelled cases; every case goes out as ONE /decide/shared request,
|
||||
every decision with "orderings": "rotations".
|
||||
|
||||
Conditions per case:
|
||||
evidence the case's real state
|
||||
Labels may be one option id or a list of acceptable ones; a case may also name "forbid"
|
||||
options per decision (an egregious pick, e.g. a joyful face on bad news), counted separately.
|
||||
blind NULL CONTROL: the same decisions over a content-free state. Whatever it still gets
|
||||
"right" is the prior the question and options carry by themselves, so the evidence
|
||||
condition is judged against it rather than against 0%.
|
||||
Positive controls are cases tagged "control" in the scenario file: unambiguous by construction,
|
||||
so a miss there means the instrument (question wording, option text) is broken, not the model.
|
||||
The service is deterministic (acceptance: same 144 rows twice, gap 0.0), so repeats measure
|
||||
latency only: RUNS full passes, median and min..max reported.
|
||||
SEMIF_URL=... SEMIF_TOKEN=... uv run --with httpx python consumer_fit.py scenario.json out.json
|
||||
"""
|
||||
import json, os, statistics as st, sys, time
|
||||
import httpx
|
||||
|
||||
U, H = os.environ["SEMIF_URL"], {"Authorization": f"Bearer {os.environ['SEMIF_TOKEN']}"}
|
||||
RUNS = int(os.environ.get("RUNS", "3"))
|
||||
BLIND = "(No evidence is available for this turn.)"
|
||||
scen = json.load(open(sys.argv[1]))
|
||||
decisions = scen["decisions"]
|
||||
|
||||
|
||||
def ask(state, only=None):
|
||||
body = {"state": state, "decisions": [
|
||||
{"id": d["id"], "question": d["question"], "options": d["options"], "orderings": "rotations"}
|
||||
for d in decisions if only is None or d["id"] in only]}
|
||||
t = time.perf_counter()
|
||||
r = httpx.post(f"{U}/decide/shared", headers=H, json=body, timeout=120)
|
||||
r.raise_for_status()
|
||||
e2e = (time.perf_counter() - t) * 1000
|
||||
j = r.json()
|
||||
out = {}
|
||||
for res in j["results"]:
|
||||
c = res["combined"]
|
||||
p = dict(zip(res["option_ids"], c["probabilities"]))
|
||||
out[res["id"]] = {"top": c["top"], "p_top": round(p[c["top"]], 3), "agreement": c["agreement"],
|
||||
"probs": {k: round(v, 3) for k, v in p.items()},
|
||||
"input_tokens": res["orderings"][0]["input_tokens"]}
|
||||
return out, e2e, j["timing"]["total_seconds"] * 1000
|
||||
|
||||
|
||||
def ok(top, label): # a label is one option id or a list of acceptable ones
|
||||
return None if label is None else top in ([label] if isinstance(label, str) else label)
|
||||
|
||||
|
||||
rows, lat = [], {"e2e_ms": [], "srv_ms": []}
|
||||
for case in scen["cases"]:
|
||||
labels, forbid = case.get("labels", {}), case.get("forbid", {})
|
||||
runs = [ask(case["state"]) for _ in range(RUNS)]
|
||||
ev = runs[0][0]
|
||||
assert all(r[0][k]["top"] == ev[k]["top"] for r in runs for k in ev), f"{case['id']}: nondeterministic top"
|
||||
for _, e2e, srv in runs:
|
||||
lat["e2e_ms"].append(e2e); lat["srv_ms"].append(srv)
|
||||
blind, _, _ = ask(BLIND)
|
||||
for d in decisions:
|
||||
k = d["id"]
|
||||
rows.append({"case": case["id"], "tag": case.get("tag", "case"), "decision": k,
|
||||
"label": labels.get(k), "top": ev[k]["top"], "p_top": ev[k]["p_top"],
|
||||
"agreement": ev[k]["agreement"], "probs": ev[k]["probs"],
|
||||
"blind_top": blind[k]["top"], "input_tokens": ev[k]["input_tokens"],
|
||||
"ok": ok(ev[k]["top"], labels.get(k)), "blind_ok": ok(blind[k]["top"], labels.get(k)),
|
||||
"egregious": ev[k]["top"] in forbid.get(k, []),
|
||||
"blind_egregious": blind[k]["top"] in forbid.get(k, [])})
|
||||
|
||||
|
||||
def acc(sub, key="ok"):
|
||||
s = [r[key] for r in sub if r[key] is not None]
|
||||
return {"n": len(s), "correct": sum(s), "acc": round(sum(s) / len(s), 3) if s else None}
|
||||
|
||||
|
||||
def med(xs):
|
||||
return {"median": round(st.median(xs), 1), "min": round(min(xs), 1), "max": round(max(xs), 1), "n": len(xs)}
|
||||
|
||||
|
||||
report = {"scenario": scen["name"], "cases": len(scen["cases"]), "runs": RUNS,
|
||||
"latency_per_turn": {k: med(v) for k, v in lat.items()},
|
||||
"by_decision": {}}
|
||||
for d in decisions:
|
||||
sub = [r for r in rows if r["decision"] == d["id"]]
|
||||
unan = [r for r in sub if r["agreement"] == 1.0]
|
||||
split = [r for r in sub if r["agreement"] < 1.0]
|
||||
report["by_decision"][d["id"]] = {
|
||||
"evidence": acc(sub), "blind_null": acc(sub, "blind_ok"),
|
||||
"controls": acc([r for r in sub if r["tag"] == "control"]),
|
||||
"non_controls": acc([r for r in sub if r["tag"] != "control"]),
|
||||
"by_label": {lab: {"evidence": acc([r for r in sub if r["label"] == lab]),
|
||||
"blind_null": acc([r for r in sub if r["label"] == lab], "blind_ok")}
|
||||
for lab in sorted({r["label"] for r in sub if isinstance(r["label"], str)})},
|
||||
"unanimous": acc(unan), "split": acc(split),
|
||||
"egregious": sum(r["egregious"] for r in sub), "blind_egregious": sum(r["blind_egregious"] for r in sub),
|
||||
"misses": [f"{r['case']}: said {r['top']} (p={r['p_top']}, agree={r['agreement']}) want {r['label']}"
|
||||
for r in sub if r["ok"] is False]}
|
||||
json.dump({"report": report, "rows": rows}, open(sys.argv[2], "w"), indent=1)
|
||||
print(json.dumps(report, indent=1))
|
||||
for r in rows:
|
||||
mark = "BAD!" if r["egregious"] else {True: "ok ", False: "MISS", None: " - "}[r["ok"]]
|
||||
print(f"{mark} {r['case']:<28} {r['decision']:<16} top={r['top']:<14} p={r['p_top']:<5} "
|
||||
f"agree={r['agreement']:<4} want={r['label']} blind={r['blind_top']}")
|
||||
@@ -0,0 +1,213 @@
|
||||
"""Wyrd scene-change spike (2026-09-27): builds consumer_fit.py scenario files.
|
||||
|
||||
Today Wyrd decides a scene change inside the GM's big `gen` structured-output calls, two ways
|
||||
(wyrd backend/wyrd_backend/gm/gen_proposer.py, orchestrator.py, story_direction.py):
|
||||
R8 reflect's plot_direction.target_node, accepted only if it is an advanceable exit
|
||||
(plot_direction.intrude is advisory and never read)
|
||||
path B raw AddStoryNode + AdvanceToNode events, gated by nothing but prose: "Add + advance to
|
||||
a NEW node ONLY when the scene moves to a genuinely different place — never every turn
|
||||
(that would churn the backdrop)." archival-memory "Watch: GM story-node churn".
|
||||
The two semif decisions that would stand in for those, per turn:
|
||||
place has the scene moved to a genuinely different place? (the path-B gate, Wyrd's own rule)
|
||||
exit which advanceable exit, if any, does this turn move into? (R8 target selection;
|
||||
options = "stay" + the real exits, so semif can only ever name a legal target)
|
||||
Semif cannot WRITE the new node (title, summary, mood) or the steering directive; the GM keeps
|
||||
that. It only makes the typed choice.
|
||||
|
||||
Story graphs are real seeds from wyrd backend/.wyrd-data/canon (read immutable): The Fletcher
|
||||
(3de8f7da916d), The Toll Road (8ad5ebeae45f), The Last Vigil (8c3da0e2915a; that campaign had
|
||||
ZERO edges, so every real scene change went through path B; one edge is added here so R8 has
|
||||
something to choose). The turn texts are authored in the campaigns' own register (their opening
|
||||
narration is real); the transcripts themselves are gone from conv-api (session_not_found).
|
||||
Labels are ONE labeller's judgment (mine). "control" cases are unambiguous.
|
||||
python wyrd_cases.py # writes consumer-fit-2026-09-27/wyrd-scene-{fletcher,toll,vigil}.json
|
||||
"""
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
GRAPHS = {
|
||||
"fletcher": {
|
||||
"current": {"title": "The Idle Wait", "summary": "You have no contracts, no coin, and nowhere to be. "
|
||||
"The town is a pressure cooker.", "location": "Oakhaven crossroads"},
|
||||
"exits": [("meet_wren", "Approach Wren", "Meeting the Fletcher",
|
||||
"You find Wren working on arrows. She has a soldier's eyes."),
|
||||
("tavern", "Go to Tavern", "Tavern Noise",
|
||||
"The local tavern is loud with soldiers from both sides.")],
|
||||
},
|
||||
"toll": {
|
||||
"current": {"title": "The First Toll", "summary": "You arrive at the tollhouse in the pouring rain, "
|
||||
"presenting your sealed letter to Sera.", "location": "north road tollhouse"},
|
||||
"exits": [("departure", "Passage Granted/Denied", "Beyond the Toll",
|
||||
"You leave the tollhouse, either with or without Sera's blessing, continuing down the north road.")],
|
||||
},
|
||||
"vigil": {
|
||||
"current": {"title": "The Last Vigil", "summary": "The player stands in the inner sanctum, guarding the "
|
||||
"Book. Lynnae enters through a hidden alcove to discuss the encroaching demons.",
|
||||
"location": "sanctum interior"},
|
||||
"exits": [("outside", "Leave the sanctum", "Outside the Sanctum",
|
||||
"Lynnae and the player step out of the interior into the exterior surroundings.")],
|
||||
},
|
||||
}
|
||||
|
||||
# (id, graph, tag, player action, character reply, place label, exit label)
|
||||
CASES = [
|
||||
# ---------------- The Fletcher: current = The Idle Wait, at the crossroads.
|
||||
("f-idle-look", "fletcher", "control", "I lean on the fence post and watch the soldiers pass.",
|
||||
"Boots churn the mud as a column of Iron Legion pikemen tramps past the crossroads, eyes forward, "
|
||||
"drums muffled by the damp. Nobody spares you a glance. Across the square, a Sylvan Pact scout watches "
|
||||
"them go from the shadow of the well, and does not move until the last of them has passed.",
|
||||
"holds", "stay"),
|
||||
("f-plan-tavern", "fletcher", "case", "Maybe I'll try the tavern later, see if anyone's hiring.",
|
||||
"A gust carries the smell of woodsmoke and spilled ale from the tavern at the far end of the square. "
|
||||
"Someone inside is laughing too loudly. It would be warm in there, and full of men with coin and "
|
||||
"grudges. For now the crossroads holds you, and the rain has not yet decided whether to fall.",
|
||||
"holds", "stay"),
|
||||
("f-go-tavern", "fletcher", "control", "I head for the tavern and push the door open.",
|
||||
"The door gives with a groan and the noise hits you like heat: Legion sergeants at one long table, "
|
||||
"Pact rangers at another, and a narrow no-man's-land of spilled ale between them. The barkeep looks "
|
||||
"up, measures your empty belt, and goes back to wiping a cup that will never be clean.",
|
||||
"moved", "tavern"),
|
||||
("f-approach-wren", "fletcher", "control", "I walk over to the woman fletching arrows by the smithy.",
|
||||
"She doesn't look up as your shadow falls across her work. Her hands keep moving, binding grey goose "
|
||||
"feather to shaft with quick, sure turns. \"These are war arrows,\" she says, \"fletched for distance "
|
||||
"and accuracy, not for some merchant's target practice. You're either very brave or very stupid.\"",
|
||||
"holds", "meet_wren"),
|
||||
("f-memory", "fletcher", "case", "I think back to the last time I held a bow, at the siege of Varn.",
|
||||
"The memory comes unbidden: the walls of Varn black against a burning sky, the bowstring biting your "
|
||||
"fingers, the screaming from the breach. You flex your hand now, in the mud of Oakhaven, and find it "
|
||||
"still remembers the draw. Nobody at the crossroads notices a ghost remembering a war.",
|
||||
"holds", "stay"),
|
||||
("f-leave-town", "fletcher", "case", "Forget this town. I take the west road out of Oakhaven.",
|
||||
"The crossroads falls behind you, then the last hovels, then the smell of the town itself. By "
|
||||
"afternoon the west road narrows between dark pines, and the only sound is your own boots and, "
|
||||
"somewhere ahead, the slow creak of a mill wheel turning in water you cannot yet see.",
|
||||
"moved", "stay"),
|
||||
("f-nightfall", "fletcher", "case", "I stay put and wait for something to happen.",
|
||||
"Hours pass. The rain finally commits, then relents. Night settles over the crossroads, and one by "
|
||||
"one the shutters close. A lantern is lit at the well. Still nothing. Still waiting. Somewhere a dog "
|
||||
"barks twice and thinks better of it.",
|
||||
"holds", "stay"),
|
||||
("f-wren-comes", "fletcher", "case", "I keep watching the square.",
|
||||
"A woman with a quiver of fresh-fletched arrows crosses the square toward you, stepping neatly around "
|
||||
"the worst of the mud. She stops an arm's length away and studies you the way a quartermaster studies "
|
||||
"a horse. \"You've been standing there all morning,\" Wren says. \"Either you're waiting for work, or "
|
||||
"you're a spy. Which is it?\"",
|
||||
"holds", "meet_wren"),
|
||||
# ---------------- The Toll Road: current = The First Toll, in the tollhouse.
|
||||
("t-reads-letter", "toll", "control", "I hand Sera the sealed letter.",
|
||||
"Sera's gloved hand emerges from the shadow to take the letter, her fingers tracing the wax seal with "
|
||||
"practiced efficiency. She turns it over slowly, examining it by the light of the single lamp. Rain "
|
||||
"hammers the iron roof. \"Whose seal is this?\" she asks, without looking up.",
|
||||
"holds", "stay"),
|
||||
("t-questions", "toll", "case", "\"A friend in the capital. That's all you need to know.\"",
|
||||
"\"That's rarely all I need to know.\" Sera sets the letter on the counter between you and leaves her "
|
||||
"hand on it. \"The road north has been kind to liars lately. It won't stay that way.\" Water drips from "
|
||||
"the eaves into a bucket by the door, counting out the silence.",
|
||||
"holds", "stay"),
|
||||
("t-future", "toll", "case", "\"Stamp it and I'll be out of your hair before the rain stops.\"",
|
||||
"\"Before the rain stops,\" Sera repeats, as if tasting the words. \"It hasn't stopped in nine days.\" "
|
||||
"She doesn't reach for the stamp. She reaches for the ledger instead, and opens it to a page thick with "
|
||||
"names, and runs one gloved finger down the column.",
|
||||
"holds", "stay"),
|
||||
("t-granted", "toll", "control", "I wait while she decides.",
|
||||
"The stamp comes down with a crack like a gavel. Sera slides the letter back and jerks her chin at the "
|
||||
"door. You step out into the downpour, and the tollhouse lamp shrinks behind you as the north road "
|
||||
"unspools ahead, black and glistening, into the hills.",
|
||||
"moved", "departure"),
|
||||
("t-denied", "toll", "case", "\"Then I'll find another way north.\" I turn to go.",
|
||||
"\"There is no other way north,\" Sera says to your back, but she lets you go. The door bangs shut "
|
||||
"behind you. You walk on anyway, past the barrier, the rain soaking through your collar, the tollhouse "
|
||||
"dwindling to a single yellow window behind you on the north road.",
|
||||
"moved", "departure"),
|
||||
("t-rider", "toll", "case", "I stay by the stove to dry off.",
|
||||
"Hooves on the road outside, then a fist on the door. A rider in a sodden Legion cloak ducks in out of "
|
||||
"the rain, glances at you, glances at Sera, and drops a heavier purse on the counter than any toll "
|
||||
"requires. Sera's hand closes over it without a word.",
|
||||
"holds", "stay"),
|
||||
("t-city-memory", "toll", "case", "\"You've never been to the capital, have you?\"",
|
||||
"Something shifts behind Sera's eyes. \"I was born there,\" she says. \"Under the bell tower, in the "
|
||||
"year the river froze. I left the day they hanged my father in the square.\" She says it flatly, the "
|
||||
"way you'd read out a toll schedule, and turns back to the ledger.",
|
||||
"holds", "stay"),
|
||||
# ---------------- The Last Vigil: current = the inner sanctum.
|
||||
("v-talk", "vigil", "control", "\"How close are they?\"",
|
||||
"Lynnae rests her fingertips on the edge of the altar, not quite touching the Book. \"Close enough that "
|
||||
"the wards are singing,\" she says. \"Listen.\" You do, and beneath the wind there is a high, thin note, "
|
||||
"like a wet finger on the rim of a glass. The candles lean toward the door.",
|
||||
"holds", "stay"),
|
||||
("v-step-out", "vigil", "control", "\"Show me.\" I follow her out through the great doors.",
|
||||
"The doors groan open and the wind takes the candle flames behind you. Outside, the sanctum steps drop "
|
||||
"away into a valley of ash, and on the far ridge a line of red lights moves slowly, patiently, like a "
|
||||
"procession of lanterns that has forgotten what it is mourning.",
|
||||
"moved", "outside"),
|
||||
("v-plan-pub", "vigil", "case", "\"If the wards fall, we take the Book to the old pub in the village.\"",
|
||||
"\"The Drowned Crown?\" Lynnae almost smiles. \"No demon would look for holy relics behind a bar.\" She "
|
||||
"turns the idea over, then nods slowly. \"If it comes to that. But the wards have not fallen yet, and I "
|
||||
"would rather they never did.\"",
|
||||
"holds", "stay"),
|
||||
("v-to-pub", "vigil", "case", "I take the Book and we run for the village pub.",
|
||||
"You run through the ash with the Book clutched to your chest, Lynnae a step behind. The heavy door "
|
||||
"creaks on rusted hinges as you cross the threshold, the air inside the tavern thick with the scent of "
|
||||
"stale ale, damp rot, and cold ash. You set the Book down on the bar.",
|
||||
"moved", "stay"),
|
||||
("v-vision", "vigil", "case", "I touch the Book's cover.",
|
||||
"The world tilts. For one breath you are elsewhere: a burning library, a hundred years ago, monks "
|
||||
"carrying armfuls of scrolls through smoke. Then the vision snaps shut like a book, and you are back in "
|
||||
"the sanctum with Lynnae's hand hard on your wrist. \"Don't,\" she says. \"Not yet.\"",
|
||||
"holds", "stay"),
|
||||
("v-demon-enters", "vigil", "case", "I draw my blade and stand before the altar.",
|
||||
"The great doors burst inward. Ash rolls across the floor, and in the middle of it something tall "
|
||||
"unfolds itself, too many joints, a crown of red light. The candles gutter out all at once. Lynnae "
|
||||
"is already chanting, her voice steady, her hands trembling.",
|
||||
"holds", "stay"),
|
||||
]
|
||||
|
||||
|
||||
def build():
|
||||
cases = []
|
||||
for cid, graph, tag, action, reply, place, exit_ in CASES:
|
||||
g = GRAPHS[graph]
|
||||
exit_opts = [{"id": "stay", "description": f"None of these yet. The current scene holds: {g['current']['title']}."}]
|
||||
exit_opts += [{"id": eid, "description": f"{label}: {title}. {summary}"} for eid, label, title, summary in g["exits"]]
|
||||
cases.append({"id": cid, "tag": tag, "graph": graph,
|
||||
"state": {"current_scene": g["current"], "player_action": action, "narration": reply},
|
||||
"labels": {"place": place, "exit": exit_,
|
||||
"place2": {"holds": "same", "moved": "elsewhere"}[place], "exit2": exit_}})
|
||||
return cases
|
||||
|
||||
|
||||
# Round 2 wordings, written AFTER round 1 failed its positive controls on `place` (5 of 6 real
|
||||
# moves read as "holds", no better than the blind null). Both rounds are reported; round 1 is
|
||||
# kept in the same request so the comparison is paired.
|
||||
def round2(g):
|
||||
loc, title = g["current"]["location"], g["current"]["title"]
|
||||
place2 = {"id": "place2", "question": f"This is one turn of an interactive story. The turn began at the "
|
||||
f"{loc}. By the end of this turn's narration, where are the characters?",
|
||||
"options": [{"id": "same", "description": f"Still at the {loc}."},
|
||||
{"id": "elsewhere", "description": f"Somewhere else: they have left the {loc} for a different place."}]}
|
||||
exit2 = {"id": "exit2", "question": "This is one turn of an interactive story. The story can move on from the "
|
||||
"current scene into one of the listed next scenes, but only once the narration actually gets there. "
|
||||
"Where does this turn leave the story?",
|
||||
"options": [{"id": "stay", "description": f"Still in the current scene, {title}: the narration has "
|
||||
"not reached any next scene yet."}]
|
||||
+ [{"id": eid, "description": f"Moved on to {t}: {summ}"} for eid, _, t, summ in g["exits"]]}
|
||||
return [place2, exit2]
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
here = Path(__file__).parent
|
||||
place = {"id": "place", "question": "This is one turn of an interactive story. Has the scene moved to a "
|
||||
"genuinely different place during this turn, or does the current scene still hold?",
|
||||
"options": [{"id": "moved", "description": "The scene moved to a genuinely different place."},
|
||||
{"id": "holds", "description": "The current scene still holds."}]}
|
||||
exit_q = ("This is one turn of an interactive story. The story can advance from the current scene into "
|
||||
"one of the listed next scenes. Which one does this turn move the story into?")
|
||||
# exit options differ per graph, so one scenario file per graph, each carrying every decision
|
||||
for graph, g in GRAPHS.items():
|
||||
cases = [c for c in build() if c["graph"] == graph]
|
||||
exit_opts = [{"id": "stay", "description": f"None of these yet. The current scene holds: {g['current']['title']}."}]
|
||||
exit_opts += [{"id": eid, "description": f"{label}: {title}. {summary}"} for eid, label, title, summary in g["exits"]]
|
||||
scen = {"name": f"wyrd-scene-{graph}",
|
||||
"decisions": [place, {"id": "exit", "question": exit_q, "options": exit_opts}] + round2(g),
|
||||
"cases": [{k: v for k, v in c.items() if k != "graph"} for c in cases]}
|
||||
(here / "consumer-fit-2026-09-27" / f"wyrd-scene-{graph}.json").write_text(json.dumps(scen, indent=1))
|
||||
Reference in New Issue
Block a user