Update: docs/specs/saga-taxonomy-redesign-plan.md — Self-critique revision: fixed claim/principle disambiguation logic, removed position-dependent rule for hypothesis/finding, added prompt assembly section, restructured decision tree as binary tree, moved bulk-update to Phase 1, added annotator guidance to validation plan, demoted speculative metadata fields to Phase 4.

docs/specs/saga-taxonomy-redesign-plan.md

@@ -2,25 +2,29 @@
 created: '2026-04-14T21:13:22+00:00'
 id: b47806bc-e187-4381-bbd2-b9d5e71ea37a
 keywords:
+- schema
+- concept
+- type
 - saga
-- concept taxonomy
+- extraction
-- extraction schema
+- types
-- MECE
+- claim
-- Rosch
+- taxonomy
-- basic-level categories
+- author
-- ontology design schools
+- principle
-- inter-annotator agreement
+- plan
-- LLM extraction
+- calibration
-- gitea-37
+- extractor
-- gitea-38
+- prompt
-- chroma
+- as
-- coherence
 links: []
-modified: '2026-04-14T21:13:22+00:00'
+modified: '2026-04-14T21:21:32+00:00'
 summary: Architecture and implementation plan for refining Saga's concept taxonomy
   and extraction schemas based on scientific research into cognitive categorization,
   ontology design, and LLM extraction best practices. Covers MECE compliance, type
-  descriptions, schema structure, and validation approach.
+  descriptions, schema structure, and validation approach. Revised to fix disambiguation
+  logic, add prompt assembly spec, restructure decision tree, and defer speculative
+  metadata.
 tags:
 - spec
 - saga
@@ -35,7 +39,7 @@ title: Saga Concept Taxonomy & Extraction Schema — Redesign Plan
 # Saga Concept Taxonomy & Extraction Schema — Redesign Plan
 
 **Author:** Atlas (Librarian & Historian)
-**Date:** 2025-07-12
+**Date:** 2025-07-12 (revised 2025-07-13)
 **Status:** Proposed — awaiting developer review
 **Tracks:** Gitea #37 (taxonomy refinement), Gitea #38 (collection architecture redesign)
 **Research basis:** Hermione `rs_concept_taxonomy_science` — 51 extracted concepts across 4 chapters
@@ -155,7 +159,7 @@ Eight schools of ontology design were identified. The two most relevant to Saga:
 
 > *"A decision tree heuristic helps annotators classify arguments through binary questions."* — Visser et al. (2021)
 
-**Implication for Saga:** The extractor prompt should include a decision tree or prioritized set of disambiguation questions. Example: "Is this a general rule the author advocates? → `principle`. Is it a structured multi-part model? → `framework`. If you can't distinguish, default to `principle`."
+**Implication for Saga:** The extractor prompt should include a binary decision tree where each fork asks a single yes/no question. The tree should front-load the most common types so the extractor reaches the right answer quickly. See §4.3 for the full specification.
 
 ### 2.5 Extraction Schema Design for LLMs (LandingAI 2025)
 
@@ -191,6 +195,7 @@ Eight schools of ontology design were identified. The two most relevant to Saga:
 | Problem | Types | Why it's not MECE | Recommended resolution |
 |---|---|---|---|
 | `claim` and `contradiction` overlap | claim, contradiction | A contradiction is a claim that targets another view. The extractor must decide: is this "the author's argument" or "the author's challenge to another argument"? Both are claims structurally. | **Rename `contradiction` to `counterargument`** (aligns with argument schema terminology used in `article`). Add to description: "A claim that explicitly targets and opposes another stated position. If the author simply states an unconventional view without targeting a specific alternative, classify as `claim`." This creates a cleaner boundary: `counterargument` requires a target; `claim` does not. |
+| `claim` and `principle` overlap | claim, principle | Both are the author putting something forward. A claim asserts what is true; a principle prescribes what to do or how to think. The boundary can be fuzzy when a claim reads like a rule ("Practice deliberate feedback"). | **Sharpen the disambiguation rule.** The test is: *can you imagine someone following this?* If yes → `principle`. If it's a statement about reality (even a strong one) → `claim`. "System 1 operates automatically" is a claim — it describes how things are. "Seek regular feedback in structured environments" is a principle — it tells you what to do. |
 | `principle` and `framework` overlap | principle, framework | A framework (e.g., "Dual Process Theory") often contains principles (e.g., "System 1 is fast and automatic"). The boundary between "a general rule" and "a structured model of thinking" depends on scale, which is subjective. | **Keep both but sharpen descriptions.** `principle` = "a single, standalone rule or guideline." `framework` = "a named, multi-component system or model that organizes multiple ideas into a structured whole." Key test: can you name it? Does it have parts? If yes → framework. If it's one rule → principle. |
 | `question` is a weak type | question | Open questions are rarely the primary knowledge atom a reader is searching for. They're more like metadata about what the author is investigating. | **Demote but keep.** Move `question` to the end of the type list (lowest priority). Consider whether it should be extracted at all, or whether it's better captured in section summaries. **Recommendation: keep for now, monitor extraction frequency.** |
 
@@ -198,7 +203,7 @@ Eight schools of ontology design were identified. The two most relevant to Saga:
 
 | Problem | Types | Why it's not MECE | Recommended resolution |
 |---|---|---|---|
-| `hypothesis` and `finding` can overlap | hypothesis, finding | A confirmed hypothesis is reported as a finding. The same statement can be both depending on where in the paper it appears (intro = hypothesis, results = finding). | **Disambiguate by position AND framing.** `hypothesis` = "a prediction or proposition being tested, stated before results." `finding` = "an observed result or outcome, stated after methodology." If the extractor can't determine position, default to `finding`. |
+| `hypothesis` and `finding` can overlap | hypothesis, finding | A confirmed hypothesis is reported as a finding. The same statement can be both depending on where in the paper it appears (intro = hypothesis, results = finding). | **Disambiguate by tense and framing, not by position.** `hypothesis` = "a prediction or proposition framed in future or conditional tense about what the study expects to find." `finding` = "an observed result framed in past tense about what the study found." Key test: is the author predicting what will happen (hypothesis) or reporting what did happen (finding)? Position in the document can be a *corroborating signal* if available, but tense/framing is the primary disambiguation criterion and works without structural metadata. |
 | `contribution` is orphaned | contribution | Contributions are often a mix of findings, methods, and frameworks. They rarely appear as standalone knowledge atoms. | **Keep but narrow.** `contribution` = "a specific claimed novelty or advance that the authors explicitly identify as their contribution, typically in the introduction or discussion." |
 
 #### `article` schema — issues found
@@ -307,35 +312,122 @@ schemas:
 
 ### 4.3 Decision Tree for Disambiguation
 
-For schemas with adjacent-type ambiguity (specifically `book`), the extractor prompt should include a decision tree. Example for `book` schema:
+For schemas with adjacent-type ambiguity (specifically `book`), the extractor prompt should include a **binary decision tree** — a series of yes/no questions where each fork leads to either a type assignment or the next question. This directly implements Visser et al.'s finding that binary decision trees improve classification consistency.
+
+The tree should front-load the most common types so the extractor reaches the right answer quickly.
+
+**Example for `book` schema:**
 
 ```
-1. Is the author defining a term or concept precisely? → DEFINITION
+Is the author asserting something as true or arguing a position?
-2. Is the author presenting a named, multi-component model? → FRAMEWORK
+  YES:
-3. Is the author stating a general rule or guideline? → PRINCIPLE
+    Is there a specific alternative view they are explicitly opposing?
-4. Is the author challenging a specific alternative view? → COUNTERARGUMENT
+      YES → COUNTERARGUMENT
-5. Is the author making any other assertion or argument? → CLAIM
+      NO → CLAIM
-6. Is the author illustrating a point with a concrete case? → EXAMPLE
+  NO:
-7. Is the author raising something they don't answer? → QUESTION
+    Is the author defining a term or concept precisely?
+      YES → DEFINITION
+      NO:
+        Is the author presenting a named, multi-component model or system?
+          YES → FRAMEWORK
+          NO:
+            Is the author stating a rule about what to do or what generally applies?
+              YES → PRINCIPLE
+              NO:
+                Is the author illustrating a point with a concrete instance?
+                  YES → EXAMPLE
+                  NO:
+                    Is the author raising something they don't answer?
+                      YES → QUESTION
+                      NO → CLAIM  (default fallback)
 ```
 
-This tree should be generated from the type priorities and embedded in the extraction prompt. It gives the LLM a deterministic fallback path when classification is ambiguous — directly implementing Visser et al.'s finding that decision trees improve inter-annotator agreement.
+**Implementation notes:**
+- This tree is generated from the `priority` ordering of types in the schema.
+- The root question targets the highest-priority type's core distinction (asserting something as true → claim, the most common and important type).
+- Each fork is a single yes/no question — the extractor never has to choose among 7 options simultaneously.
+- The final fallback is always the schema's `default_type`.
+- For schemas without significant ambiguity (e.g., `legal_document`), a decision tree is unnecessary — the standard type list suffices.
 
-### 4.4 Concept Metadata Schema Additions
+### 4.4 Prompt Assembly
 
-Add to the concept metadata in `config.yaml`:
+The enhanced type specifications must be assembled into the actual LLM extraction prompt. This section specifies how.
+
+**Which fields go into the prompt, and in what order:**
+
+For each concept type in the schema, the prompt should include:
+
+1. `label` — bolded header
+2. `long_description` — full paragraph
+3. `linguistic_cues` — bullet list (if present)
+4. `disambiguation` — bullet list (if present)
+5. `prototypical_examples` — numbered list (1-2 examples, not all)
+
+**Fields NOT included in the prompt:** `short_description` (for human consumption only), `priority` (used for ordering only).
+
+**Token budget guidance:**
+
+A typical concept type spec assembled as above runs ~150-250 tokens. With 6-7 types per schema, that's ~1,000-1,750 tokens for the type definitions section — well within a standard extraction prompt context. If schemas grow beyond 10 types, consider using `short_description` instead of `long_description` and dropping `linguistic_cues` to stay within budget.
+
+**Decision tree placement:**
+
+If a decision tree is used (§4.3), it appears *after* the type definitions and *before* the extraction instructions. The prompt structure should be:
+
+```
+[Schema scope statement]
+[Type definitions — ordered by priority]
+[Decision tree for disambiguation]
+[Extraction instructions — format, output schema, constraints]
+```
+
+**Example assembled prompt fragment for `book.claim`:**
+
+```
+## Claim
+A declarative statement that the author argues for, defends, or puts forward as true.
+This is the author's voice making a substantive case. Claims are the primary knowledge
+atoms in non-fiction — they are what a reader would quote or reference when discussing
+the book's arguments.
+
+Signals: "I argue that...", "The evidence shows...", "It is clear that...", declarative
+sentences asserting something as true.
+
+Not to be confused with: definitions (explaining what a term means), cited facts (evidence
+from elsewhere), challenges to specific alternatives (counterarguments), or prescriptive
+rules (principles).
+
+Examples:
+1. "System 1 operates automatically and quickly with little or no effort and no sense of
+   voluntary control."
+2. "Expert intuition is more reliable than we think, but only in environments with regular
+   feedback."
+```
+
+### 4.5 Concept Metadata Schema Additions
+
+**Phase 1 — essential addition:**
 
 ```yaml
 concept_metadata:
   # Existing fields preserved...
   
-  # New fields:
+  # New field:
-  extraction_rationale:    # Short explanation of why this type was chosen (string, optional)
+  source_position:         # Where in the document structure this appeared
-  adjacent_type_score:     # If the extractor considered a second type, record it (string, optional)
+                           # (string: "introduction", "body", "conclusion", "methodology", "results")
-  source_position:         # Where in the document structure this appeared (string: "introduction", "body", "conclusion", "methodology", "results")
 ```
 
-**Why:** `extraction_rationale` and `adjacent_type_score` give us the data we need to measure classification ambiguity without re-reading the source. `source_position` enables position-based disambiguation (e.g., distinguishing hypothesis from finding in academic papers).
+`source_position` enables position-aware disambiguation as a corroborating signal (e.g., in `academic_paper`, statements in a "results" section are more likely to be `finding`). It is optional in Phase 1 — populated when the section structure is available from the parsed document.
+
+**Phase 4 — optional diagnostic fields:**
+
+These are deferred to Phase 4 and included only if calibration reveals systematic confusion that needs debugging:
+
+```yaml
+  extraction_rationale:    # Short explanation of why this type was chosen (string, optional)
+  adjacent_type_score:     # If the extractor considered a second type, record it (string, optional)
+```
+
+**Why these are deferred:** `extraction_rationale` adds prompt overhead for every extracted concept, and LLMs are known to confabulate post-hoc justifications rather than accurately reporting their reasoning. `adjacent_type_score` asks the LLM to quantify its own uncertainty — something LLMs are notably poor at. Neither field is used by the validation plan (§5), which compares against hand labels. Their value is purely for post-hoc debugging of systematic confusion patterns. They should only be added if Phase 2-3 calibration reveals a specific problem they would help diagnose.
 
 ---
 
@@ -345,9 +437,15 @@ concept_metadata:
 
 Create a gold-standard calibration dataset:
 
-1. **Select 3-5 representative documents** (one per major schema: book, academic_paper, article, plus one wild card)
+1. **Select 3-5 representative documents** — prefer documents that have already been ingested so we can compare existing extractions directly against the hand annotations. One per major schema: book, academic_paper, article, plus one wild card.
-2. **Hand-annotate 50-100 concepts per document** with type assignments
+
-3. **Record disambiguation notes** (why this was classified as X and not Y)
+2. **Hand-annotate 50-100 concepts per document** with type assignments, using the following process:
+   - **Primary annotator:** Atlas (the Librarian), working from the source text and the enhanced type specifications.
+   - **Annotation protocol:** For each concept, record the hand label, rationale, and any adjacent type considered. Use the same JSON schema the extractor produces so comparison is direct.
+   - **Inter-annotator check:** After the primary annotation is complete, a second annotator (the user or another agent) independently labels a 20% subset (10-20 concepts per document). Measure human-human agreement on this subset. If human-human agreement is below 90%, the type definitions themselves may be ambiguous and need sharpening *before* measuring LLM accuracy. This addresses the meta-problem Visser et al. identified: if humans can't agree, the taxonomy is the problem, not the annotator.
+
+3. **Record disambiguation notes** — for every concept where the annotator hesitated or considered an alternative type, record why. These notes are the raw material for future description improvements.
+
 4. **Store in `validation/calibration/`** as JSON files with schema:
    ```json
    {
@@ -355,9 +453,9 @@ Create a gold-standard calibration dataset:
      "section": "Ch. 1, Sec. 2",
      "verbatim": "System 1 operates automatically...",
      "hand_label": "claim",
-     "rationale": "Author's direct assertion, not a definition",
+     "rationale": "Author's direct assertion about how cognition works",
      "adjacent_considered": "definition",
-     "notes": "Could be confused with definition because it describes System 1"
+     "notes": "Could be confused with definition because it describes System 1, but it's asserting a factual claim, not explaining a term"
    }
    ```
 
@@ -395,18 +493,20 @@ This is not a one-time exercise. As new source types are added or existing schem
 
 ### Phase 1 — Foundation (config.yaml changes)
 
-**Goal:** Upgrade the type specification format without changing any type names or boundaries yet.
+**Goal:** Upgrade the type specification format and rename one type, without changing any structural boundaries.
 
 **Tasks:**
 1. [ ] Add `scope` field to each of the 7 schemas in `config.yaml`
 2. [ ] Add `long_description`, `linguistic_cues`, `disambiguation`, `prototypical_examples`, `priority` to each concept type in each schema
 3. [ ] Rename `contradiction` → `counterargument` in `book` schema
 4. [ ] Change `instruction_manual` default type from `tip` to `step`
-5. [ ] Add new metadata fields: `extraction_rationale`, `adjacent_type_score`, `source_position`
+5. [ ] Add `source_position` to concept metadata (optional — populated when section structure is available)
-6. [ ] Update extractor prompt template to include decision tree for schemas with adjacent-type ambiguity
+6. [ ] Update extractor prompt template to include binary decision tree for schemas with adjacent-type ambiguity (book, academic_paper, article)
-7. [ ] Update `docs/specs/concept-taxonomy-reference.md` to reflect all changes
+7. [ ] Update prompt assembly logic to include `long_description`, `linguistic_cues`, `disambiguation`, and `prototypical_examples` for each type (see §4.4)
+8. [ ] Bulk-update any existing `contradiction`-typed concepts in Chroma to `counterargument` (conditional — only if such concepts exist)
+9. [ ] Update `docs/specs/concept-taxonomy-reference.md` to reflect all changes
 
-**Deliverable:** Updated `config.yaml` with enhanced type specs, updated reference doc.
+**Deliverable:** Updated `config.yaml` with enhanced type specs, updated prompt template, updated reference doc, migrated legacy concepts.
 
 **Estimated effort:** Small — mostly YAML editing and prompt engineering.
 
@@ -416,7 +516,7 @@ This is not a one-time exercise. As new source types are added or existing schem
 
 **Tasks:**
 1. [ ] Create `validation/calibration/` directory structure
-2. [ ] Hand-annotate calibration dataset (3-5 documents, 50-100 concepts each)
+2. [ ] Hand-annotate calibration dataset (3-5 documents, 50-100 concepts each) — primary annotation by Atlas, 20% subset verified by a second annotator
 3. [ ] Build comparison script: `scripts/validate_extraction.py` that takes extractor output and calibration data, produces confusion matrix and per-type metrics
 4. [ ] Run baseline measurement with current extractor against calibration dataset (before Phase 1 changes, if possible — otherwise against Phase 1 output)
 5. [ ] Document baseline metrics in `validation/baseline_metrics.md`
@@ -448,7 +548,8 @@ This is not a one-time exercise. As new source types are added or existing schem
 **Tasks:**
 1. [ ] If adding new source types, use the enhanced type specification format from Phase 1
 2. [ ] Add calibration entries for new source types before deploying
-3. [ ] Consider adding `source_position`-based extraction rules (e.g., in `academic_paper`, weight `hypothesis` higher in introduction sections, `finding` higher in results sections)
+3. [ ] Consider adding `source_position`-based extraction rules (e.g., in `academic_paper`, weight `hypothesis` higher in introduction sections, `finding` higher in results sections) — now that we have calibration data to measure whether position-awareness improves accuracy
+4. [ ] If calibration revealed systematic confusion that needs deeper diagnosis, add `extraction_rationale` and `adjacent_type_score` as optional metadata fields — but only if there's a specific problem to solve
 
 **Deliverable:** Extension-ready architecture.
 
@@ -458,13 +559,9 @@ This is not a one-time exercise. As new source types are added or existing schem
 
 These are decisions that should be made before or during Phase 1, but don't block starting the work:
 
-1. **Should we add `source_position` to the extraction pipeline?** This requires the extractor to know which section of the document it's processing (introduction vs. body vs. conclusion). Currently, concepts are extracted per-section, and section context is in the prompt, but it's not explicitly labeled as a structural position. Adding it would help with hypothesis/finding disambiguation in academic papers.
+1. **Should we add `source_position` as a required extraction field?** Currently proposed as optional in Phase 1. Making it required means every extraction includes section-aware position, which helps with hypothesis/finding disambiguation in academic papers. But it depends on the parsed document providing section labels consistently. Recommendation: start optional, require it if Phase 2 calibration shows position would have resolved confusions.
 
-2. **Should `extraction_rationale` be stored in Chroma metadata or only in the extraction JSON?** Storing it in Chroma adds useful debugging data but increases storage per concept. Recommendation: store in extraction JSON, not in Chroma.
+2. **What's the threshold for creating a new schema vs. extending an existing one?** If we ingest a new document type (e.g., `podcast_transcript`), should it get its own schema or use the closest existing one? Recommendation: start by mapping to the closest existing schema, and only create a new schema if calibration shows ≥30% of concepts don't fit any existing type cleanly.
-
-3. **Should we re-index existing ingested concepts after type changes?** If we rename `contradiction` → `counterargument`, existing concepts tagged `contradiction` will be orphaned from the new taxonomy. Options: (a) bulk update the type label in Chroma, (b) leave them as-is and accept the inconsistency, (c) re-extract from source. Recommendation: (a) bulk update — it's a simple rename.
-
-4. **What's the threshold for creating a new schema vs. extending an existing one?** If we ingest a new document type (e.g., `podcast_transcript`), should it get its own schema or use the closest existing one? Recommendation: start by mapping to the closest existing schema, and only create a new schema if calibration shows ≥30% of concepts don't fit any existing type cleanly.
 
 ---
 
@@ -502,10 +599,13 @@ To be explicit about stability:
 | `instruction_manual` default | `tip` | Change to `step` | 1 |
 | Type descriptions | 1-sentence label | Structured spec (long desc + cues + disambiguation + examples + priority) | 1 |
 | Schema scope | Implicit (schema name) | Explicit `scope` field | 1 |
-| Extractor prompt | Type list only | Type list + decision tree for ambiguous schemas | 1 |
+| Extractor prompt | Type list only | Type list + binary decision tree for ambiguous schemas | 1 |
-| Concept metadata | Current fields | Add `extraction_rationale`, `adjacent_type_score`, `source_position` | 1 |
+| Prompt assembly | Not specified | Explicit field ordering and token budget guidance (§4.4) | 1 |
+| Concept metadata | Current fields | Add `source_position` (optional) | 1 |
+| Legacy data migration | Not addressed | Bulk-update `contradiction` → `counterargument` in Chroma | 1 |
 | Validation infrastructure | None | Calibration dataset + comparison script + baseline metrics | 2 |
 | Quality target | Not measured | ≥85% type agreement vs. hand labels | 3 |
+| Diagnostic metadata | Not applicable | `extraction_rationale`, `adjacent_type_score` — only if calibration reveals need | 4 |
 
 ## Appendix B: Example Enhanced Type Spec — `book.claim`
 
@@ -529,7 +629,7 @@ claim:
     - "NOT a definition of a term (use 'definition' — definitions explain what something means, claims assert that something is true)"
     - "NOT a cited external fact (use 'evidence' — claims are the author's own, evidence is borrowed)"
     - "NOT a challenge to a specific alternative (use 'counterargument' — counterarguments explicitly target another view)"
-    - "NOT a general rule without argumentative force (use 'principle' — principles are prescriptive, claims are descriptive)"
+    - "NOT a prescriptive rule or guideline (use 'principle' — principles tell you what to do or what generally applies, claims describe what is true)"
   prototypical_examples:
     - "System 1 operates automatically and quickly with little or no effort and no sense of voluntary control."
     - "The planning fallacy describes a tendency for people to underestimate the time, costs, and risks of future actions."