diff --git a/docs/specs/saga-taxonomy-redesign-plan.md b/docs/specs/saga-taxonomy-redesign-plan.md index bd624e6..40b84ad 100644 --- a/docs/specs/saga-taxonomy-redesign-plan.md +++ b/docs/specs/saga-taxonomy-redesign-plan.md @@ -2,25 +2,29 @@ created: '2026-04-14T21:13:22+00:00' id: b47806bc-e187-4381-bbd2-b9d5e71ea37a keywords: +- schema +- concept +- type - saga -- concept taxonomy -- extraction schema -- MECE -- Rosch -- basic-level categories -- ontology design schools -- inter-annotator agreement -- LLM extraction -- gitea-37 -- gitea-38 -- chroma -- coherence +- extraction +- types +- claim +- taxonomy +- author +- principle +- plan +- calibration +- extractor +- prompt +- as links: [] -modified: '2026-04-14T21:13:22+00:00' +modified: '2026-04-14T21:21:32+00:00' summary: Architecture and implementation plan for refining Saga's concept taxonomy and extraction schemas based on scientific research into cognitive categorization, ontology design, and LLM extraction best practices. Covers MECE compliance, type - descriptions, schema structure, and validation approach. + descriptions, schema structure, and validation approach. Revised to fix disambiguation + logic, add prompt assembly spec, restructure decision tree, and defer speculative + metadata. tags: - spec - saga @@ -35,7 +39,7 @@ title: Saga Concept Taxonomy & Extraction Schema — Redesign Plan # Saga Concept Taxonomy & Extraction Schema — Redesign Plan **Author:** Atlas (Librarian & Historian) -**Date:** 2025-07-12 +**Date:** 2025-07-12 (revised 2025-07-13) **Status:** Proposed — awaiting developer review **Tracks:** Gitea #37 (taxonomy refinement), Gitea #38 (collection architecture redesign) **Research basis:** Hermione `rs_concept_taxonomy_science` — 51 extracted concepts across 4 chapters @@ -155,7 +159,7 @@ Eight schools of ontology design were identified. The two most relevant to Saga: > *"A decision tree heuristic helps annotators classify arguments through binary questions."* — Visser et al. (2021) -**Implication for Saga:** The extractor prompt should include a decision tree or prioritized set of disambiguation questions. Example: "Is this a general rule the author advocates? → `principle`. Is it a structured multi-part model? → `framework`. If you can't distinguish, default to `principle`." +**Implication for Saga:** The extractor prompt should include a binary decision tree where each fork asks a single yes/no question. The tree should front-load the most common types so the extractor reaches the right answer quickly. See §4.3 for the full specification. ### 2.5 Extraction Schema Design for LLMs (LandingAI 2025) @@ -191,6 +195,7 @@ Eight schools of ontology design were identified. The two most relevant to Saga: | Problem | Types | Why it's not MECE | Recommended resolution | |---|---|---|---| | `claim` and `contradiction` overlap | claim, contradiction | A contradiction is a claim that targets another view. The extractor must decide: is this "the author's argument" or "the author's challenge to another argument"? Both are claims structurally. | **Rename `contradiction` to `counterargument`** (aligns with argument schema terminology used in `article`). Add to description: "A claim that explicitly targets and opposes another stated position. If the author simply states an unconventional view without targeting a specific alternative, classify as `claim`." This creates a cleaner boundary: `counterargument` requires a target; `claim` does not. | +| `claim` and `principle` overlap | claim, principle | Both are the author putting something forward. A claim asserts what is true; a principle prescribes what to do or how to think. The boundary can be fuzzy when a claim reads like a rule ("Practice deliberate feedback"). | **Sharpen the disambiguation rule.** The test is: *can you imagine someone following this?* If yes → `principle`. If it's a statement about reality (even a strong one) → `claim`. "System 1 operates automatically" is a claim — it describes how things are. "Seek regular feedback in structured environments" is a principle — it tells you what to do. | | `principle` and `framework` overlap | principle, framework | A framework (e.g., "Dual Process Theory") often contains principles (e.g., "System 1 is fast and automatic"). The boundary between "a general rule" and "a structured model of thinking" depends on scale, which is subjective. | **Keep both but sharpen descriptions.** `principle` = "a single, standalone rule or guideline." `framework` = "a named, multi-component system or model that organizes multiple ideas into a structured whole." Key test: can you name it? Does it have parts? If yes → framework. If it's one rule → principle. | | `question` is a weak type | question | Open questions are rarely the primary knowledge atom a reader is searching for. They're more like metadata about what the author is investigating. | **Demote but keep.** Move `question` to the end of the type list (lowest priority). Consider whether it should be extracted at all, or whether it's better captured in section summaries. **Recommendation: keep for now, monitor extraction frequency.** | @@ -198,7 +203,7 @@ Eight schools of ontology design were identified. The two most relevant to Saga: | Problem | Types | Why it's not MECE | Recommended resolution | |---|---|---|---| -| `hypothesis` and `finding` can overlap | hypothesis, finding | A confirmed hypothesis is reported as a finding. The same statement can be both depending on where in the paper it appears (intro = hypothesis, results = finding). | **Disambiguate by position AND framing.** `hypothesis` = "a prediction or proposition being tested, stated before results." `finding` = "an observed result or outcome, stated after methodology." If the extractor can't determine position, default to `finding`. | +| `hypothesis` and `finding` can overlap | hypothesis, finding | A confirmed hypothesis is reported as a finding. The same statement can be both depending on where in the paper it appears (intro = hypothesis, results = finding). | **Disambiguate by tense and framing, not by position.** `hypothesis` = "a prediction or proposition framed in future or conditional tense about what the study expects to find." `finding` = "an observed result framed in past tense about what the study found." Key test: is the author predicting what will happen (hypothesis) or reporting what did happen (finding)? Position in the document can be a *corroborating signal* if available, but tense/framing is the primary disambiguation criterion and works without structural metadata. | | `contribution` is orphaned | contribution | Contributions are often a mix of findings, methods, and frameworks. They rarely appear as standalone knowledge atoms. | **Keep but narrow.** `contribution` = "a specific claimed novelty or advance that the authors explicitly identify as their contribution, typically in the introduction or discussion." | #### `article` schema — issues found @@ -307,35 +312,122 @@ schemas: ### 4.3 Decision Tree for Disambiguation -For schemas with adjacent-type ambiguity (specifically `book`), the extractor prompt should include a decision tree. Example for `book` schema: +For schemas with adjacent-type ambiguity (specifically `book`), the extractor prompt should include a **binary decision tree** — a series of yes/no questions where each fork leads to either a type assignment or the next question. This directly implements Visser et al.'s finding that binary decision trees improve classification consistency. + +The tree should front-load the most common types so the extractor reaches the right answer quickly. + +**Example for `book` schema:** ``` -1. Is the author defining a term or concept precisely? → DEFINITION -2. Is the author presenting a named, multi-component model? → FRAMEWORK -3. Is the author stating a general rule or guideline? → PRINCIPLE -4. Is the author challenging a specific alternative view? → COUNTERARGUMENT -5. Is the author making any other assertion or argument? → CLAIM -6. Is the author illustrating a point with a concrete case? → EXAMPLE -7. Is the author raising something they don't answer? → QUESTION +Is the author asserting something as true or arguing a position? + YES: + Is there a specific alternative view they are explicitly opposing? + YES → COUNTERARGUMENT + NO → CLAIM + NO: + Is the author defining a term or concept precisely? + YES → DEFINITION + NO: + Is the author presenting a named, multi-component model or system? + YES → FRAMEWORK + NO: + Is the author stating a rule about what to do or what generally applies? + YES → PRINCIPLE + NO: + Is the author illustrating a point with a concrete instance? + YES → EXAMPLE + NO: + Is the author raising something they don't answer? + YES → QUESTION + NO → CLAIM (default fallback) ``` -This tree should be generated from the type priorities and embedded in the extraction prompt. It gives the LLM a deterministic fallback path when classification is ambiguous — directly implementing Visser et al.'s finding that decision trees improve inter-annotator agreement. +**Implementation notes:** +- This tree is generated from the `priority` ordering of types in the schema. +- The root question targets the highest-priority type's core distinction (asserting something as true → claim, the most common and important type). +- Each fork is a single yes/no question — the extractor never has to choose among 7 options simultaneously. +- The final fallback is always the schema's `default_type`. +- For schemas without significant ambiguity (e.g., `legal_document`), a decision tree is unnecessary — the standard type list suffices. -### 4.4 Concept Metadata Schema Additions +### 4.4 Prompt Assembly -Add to the concept metadata in `config.yaml`: +The enhanced type specifications must be assembled into the actual LLM extraction prompt. This section specifies how. + +**Which fields go into the prompt, and in what order:** + +For each concept type in the schema, the prompt should include: + +1. `label` — bolded header +2. `long_description` — full paragraph +3. `linguistic_cues` — bullet list (if present) +4. `disambiguation` — bullet list (if present) +5. `prototypical_examples` — numbered list (1-2 examples, not all) + +**Fields NOT included in the prompt:** `short_description` (for human consumption only), `priority` (used for ordering only). + +**Token budget guidance:** + +A typical concept type spec assembled as above runs ~150-250 tokens. With 6-7 types per schema, that's ~1,000-1,750 tokens for the type definitions section — well within a standard extraction prompt context. If schemas grow beyond 10 types, consider using `short_description` instead of `long_description` and dropping `linguistic_cues` to stay within budget. + +**Decision tree placement:** + +If a decision tree is used (§4.3), it appears *after* the type definitions and *before* the extraction instructions. The prompt structure should be: + +``` +[Schema scope statement] +[Type definitions — ordered by priority] +[Decision tree for disambiguation] +[Extraction instructions — format, output schema, constraints] +``` + +**Example assembled prompt fragment for `book.claim`:** + +``` +## Claim +A declarative statement that the author argues for, defends, or puts forward as true. +This is the author's voice making a substantive case. Claims are the primary knowledge +atoms in non-fiction — they are what a reader would quote or reference when discussing +the book's arguments. + +Signals: "I argue that...", "The evidence shows...", "It is clear that...", declarative +sentences asserting something as true. + +Not to be confused with: definitions (explaining what a term means), cited facts (evidence +from elsewhere), challenges to specific alternatives (counterarguments), or prescriptive +rules (principles). + +Examples: +1. "System 1 operates automatically and quickly with little or no effort and no sense of + voluntary control." +2. "Expert intuition is more reliable than we think, but only in environments with regular + feedback." +``` + +### 4.5 Concept Metadata Schema Additions + +**Phase 1 — essential addition:** ```yaml concept_metadata: # Existing fields preserved... - # New fields: - extraction_rationale: # Short explanation of why this type was chosen (string, optional) - adjacent_type_score: # If the extractor considered a second type, record it (string, optional) - source_position: # Where in the document structure this appeared (string: "introduction", "body", "conclusion", "methodology", "results") + # New field: + source_position: # Where in the document structure this appeared + # (string: "introduction", "body", "conclusion", "methodology", "results") ``` -**Why:** `extraction_rationale` and `adjacent_type_score` give us the data we need to measure classification ambiguity without re-reading the source. `source_position` enables position-based disambiguation (e.g., distinguishing hypothesis from finding in academic papers). +`source_position` enables position-aware disambiguation as a corroborating signal (e.g., in `academic_paper`, statements in a "results" section are more likely to be `finding`). It is optional in Phase 1 — populated when the section structure is available from the parsed document. + +**Phase 4 — optional diagnostic fields:** + +These are deferred to Phase 4 and included only if calibration reveals systematic confusion that needs debugging: + +```yaml + extraction_rationale: # Short explanation of why this type was chosen (string, optional) + adjacent_type_score: # If the extractor considered a second type, record it (string, optional) +``` + +**Why these are deferred:** `extraction_rationale` adds prompt overhead for every extracted concept, and LLMs are known to confabulate post-hoc justifications rather than accurately reporting their reasoning. `adjacent_type_score` asks the LLM to quantify its own uncertainty — something LLMs are notably poor at. Neither field is used by the validation plan (§5), which compares against hand labels. Their value is purely for post-hoc debugging of systematic confusion patterns. They should only be added if Phase 2-3 calibration reveals a specific problem they would help diagnose. --- @@ -345,9 +437,15 @@ concept_metadata: Create a gold-standard calibration dataset: -1. **Select 3-5 representative documents** (one per major schema: book, academic_paper, article, plus one wild card) -2. **Hand-annotate 50-100 concepts per document** with type assignments -3. **Record disambiguation notes** (why this was classified as X and not Y) +1. **Select 3-5 representative documents** — prefer documents that have already been ingested so we can compare existing extractions directly against the hand annotations. One per major schema: book, academic_paper, article, plus one wild card. + +2. **Hand-annotate 50-100 concepts per document** with type assignments, using the following process: + - **Primary annotator:** Atlas (the Librarian), working from the source text and the enhanced type specifications. + - **Annotation protocol:** For each concept, record the hand label, rationale, and any adjacent type considered. Use the same JSON schema the extractor produces so comparison is direct. + - **Inter-annotator check:** After the primary annotation is complete, a second annotator (the user or another agent) independently labels a 20% subset (10-20 concepts per document). Measure human-human agreement on this subset. If human-human agreement is below 90%, the type definitions themselves may be ambiguous and need sharpening *before* measuring LLM accuracy. This addresses the meta-problem Visser et al. identified: if humans can't agree, the taxonomy is the problem, not the annotator. + +3. **Record disambiguation notes** — for every concept where the annotator hesitated or considered an alternative type, record why. These notes are the raw material for future description improvements. + 4. **Store in `validation/calibration/`** as JSON files with schema: ```json { @@ -355,9 +453,9 @@ Create a gold-standard calibration dataset: "section": "Ch. 1, Sec. 2", "verbatim": "System 1 operates automatically...", "hand_label": "claim", - "rationale": "Author's direct assertion, not a definition", + "rationale": "Author's direct assertion about how cognition works", "adjacent_considered": "definition", - "notes": "Could be confused with definition because it describes System 1" + "notes": "Could be confused with definition because it describes System 1, but it's asserting a factual claim, not explaining a term" } ``` @@ -395,18 +493,20 @@ This is not a one-time exercise. As new source types are added or existing schem ### Phase 1 — Foundation (config.yaml changes) -**Goal:** Upgrade the type specification format without changing any type names or boundaries yet. +**Goal:** Upgrade the type specification format and rename one type, without changing any structural boundaries. **Tasks:** 1. [ ] Add `scope` field to each of the 7 schemas in `config.yaml` 2. [ ] Add `long_description`, `linguistic_cues`, `disambiguation`, `prototypical_examples`, `priority` to each concept type in each schema 3. [ ] Rename `contradiction` → `counterargument` in `book` schema 4. [ ] Change `instruction_manual` default type from `tip` to `step` -5. [ ] Add new metadata fields: `extraction_rationale`, `adjacent_type_score`, `source_position` -6. [ ] Update extractor prompt template to include decision tree for schemas with adjacent-type ambiguity -7. [ ] Update `docs/specs/concept-taxonomy-reference.md` to reflect all changes +5. [ ] Add `source_position` to concept metadata (optional — populated when section structure is available) +6. [ ] Update extractor prompt template to include binary decision tree for schemas with adjacent-type ambiguity (book, academic_paper, article) +7. [ ] Update prompt assembly logic to include `long_description`, `linguistic_cues`, `disambiguation`, and `prototypical_examples` for each type (see §4.4) +8. [ ] Bulk-update any existing `contradiction`-typed concepts in Chroma to `counterargument` (conditional — only if such concepts exist) +9. [ ] Update `docs/specs/concept-taxonomy-reference.md` to reflect all changes -**Deliverable:** Updated `config.yaml` with enhanced type specs, updated reference doc. +**Deliverable:** Updated `config.yaml` with enhanced type specs, updated prompt template, updated reference doc, migrated legacy concepts. **Estimated effort:** Small — mostly YAML editing and prompt engineering. @@ -416,7 +516,7 @@ This is not a one-time exercise. As new source types are added or existing schem **Tasks:** 1. [ ] Create `validation/calibration/` directory structure -2. [ ] Hand-annotate calibration dataset (3-5 documents, 50-100 concepts each) +2. [ ] Hand-annotate calibration dataset (3-5 documents, 50-100 concepts each) — primary annotation by Atlas, 20% subset verified by a second annotator 3. [ ] Build comparison script: `scripts/validate_extraction.py` that takes extractor output and calibration data, produces confusion matrix and per-type metrics 4. [ ] Run baseline measurement with current extractor against calibration dataset (before Phase 1 changes, if possible — otherwise against Phase 1 output) 5. [ ] Document baseline metrics in `validation/baseline_metrics.md` @@ -448,7 +548,8 @@ This is not a one-time exercise. As new source types are added or existing schem **Tasks:** 1. [ ] If adding new source types, use the enhanced type specification format from Phase 1 2. [ ] Add calibration entries for new source types before deploying -3. [ ] Consider adding `source_position`-based extraction rules (e.g., in `academic_paper`, weight `hypothesis` higher in introduction sections, `finding` higher in results sections) +3. [ ] Consider adding `source_position`-based extraction rules (e.g., in `academic_paper`, weight `hypothesis` higher in introduction sections, `finding` higher in results sections) — now that we have calibration data to measure whether position-awareness improves accuracy +4. [ ] If calibration revealed systematic confusion that needs deeper diagnosis, add `extraction_rationale` and `adjacent_type_score` as optional metadata fields — but only if there's a specific problem to solve **Deliverable:** Extension-ready architecture. @@ -458,13 +559,9 @@ This is not a one-time exercise. As new source types are added or existing schem These are decisions that should be made before or during Phase 1, but don't block starting the work: -1. **Should we add `source_position` to the extraction pipeline?** This requires the extractor to know which section of the document it's processing (introduction vs. body vs. conclusion). Currently, concepts are extracted per-section, and section context is in the prompt, but it's not explicitly labeled as a structural position. Adding it would help with hypothesis/finding disambiguation in academic papers. +1. **Should we add `source_position` as a required extraction field?** Currently proposed as optional in Phase 1. Making it required means every extraction includes section-aware position, which helps with hypothesis/finding disambiguation in academic papers. But it depends on the parsed document providing section labels consistently. Recommendation: start optional, require it if Phase 2 calibration shows position would have resolved confusions. -2. **Should `extraction_rationale` be stored in Chroma metadata or only in the extraction JSON?** Storing it in Chroma adds useful debugging data but increases storage per concept. Recommendation: store in extraction JSON, not in Chroma. - -3. **Should we re-index existing ingested concepts after type changes?** If we rename `contradiction` → `counterargument`, existing concepts tagged `contradiction` will be orphaned from the new taxonomy. Options: (a) bulk update the type label in Chroma, (b) leave them as-is and accept the inconsistency, (c) re-extract from source. Recommendation: (a) bulk update — it's a simple rename. - -4. **What's the threshold for creating a new schema vs. extending an existing one?** If we ingest a new document type (e.g., `podcast_transcript`), should it get its own schema or use the closest existing one? Recommendation: start by mapping to the closest existing schema, and only create a new schema if calibration shows ≥30% of concepts don't fit any existing type cleanly. +2. **What's the threshold for creating a new schema vs. extending an existing one?** If we ingest a new document type (e.g., `podcast_transcript`), should it get its own schema or use the closest existing one? Recommendation: start by mapping to the closest existing schema, and only create a new schema if calibration shows ≥30% of concepts don't fit any existing type cleanly. --- @@ -502,10 +599,13 @@ To be explicit about stability: | `instruction_manual` default | `tip` | Change to `step` | 1 | | Type descriptions | 1-sentence label | Structured spec (long desc + cues + disambiguation + examples + priority) | 1 | | Schema scope | Implicit (schema name) | Explicit `scope` field | 1 | -| Extractor prompt | Type list only | Type list + decision tree for ambiguous schemas | 1 | -| Concept metadata | Current fields | Add `extraction_rationale`, `adjacent_type_score`, `source_position` | 1 | +| Extractor prompt | Type list only | Type list + binary decision tree for ambiguous schemas | 1 | +| Prompt assembly | Not specified | Explicit field ordering and token budget guidance (§4.4) | 1 | +| Concept metadata | Current fields | Add `source_position` (optional) | 1 | +| Legacy data migration | Not addressed | Bulk-update `contradiction` → `counterargument` in Chroma | 1 | | Validation infrastructure | None | Calibration dataset + comparison script + baseline metrics | 2 | | Quality target | Not measured | ≥85% type agreement vs. hand labels | 3 | +| Diagnostic metadata | Not applicable | `extraction_rationale`, `adjacent_type_score` — only if calibration reveals need | 4 | ## Appendix B: Example Enhanced Type Spec — `book.claim` @@ -529,7 +629,7 @@ claim: - "NOT a definition of a term (use 'definition' — definitions explain what something means, claims assert that something is true)" - "NOT a cited external fact (use 'evidence' — claims are the author's own, evidence is borrowed)" - "NOT a challenge to a specific alternative (use 'counterargument' — counterarguments explicitly target another view)" - - "NOT a general rule without argumentative force (use 'principle' — principles are prescriptive, claims are descriptive)" + - "NOT a prescriptive rule or guideline (use 'principle' — principles tell you what to do or what generally applies, claims describe what is true)" prototypical_examples: - "System 1 operates automatically and quickly with little or no effort and no sense of voluntary control." - "The planning fallacy describes a tendency for people to underestimate the time, costs, and risks of future actions."