diff --git a/docs/specs/saga-taxonomy-redesign-plan.md b/docs/specs/saga-taxonomy-redesign-plan.md new file mode 100644 index 0000000..bd624e6 --- /dev/null +++ b/docs/specs/saga-taxonomy-redesign-plan.md @@ -0,0 +1,538 @@ +--- +created: '2026-04-14T21:13:22+00:00' +id: b47806bc-e187-4381-bbd2-b9d5e71ea37a +keywords: +- saga +- concept taxonomy +- extraction schema +- MECE +- Rosch +- basic-level categories +- ontology design schools +- inter-annotator agreement +- LLM extraction +- gitea-37 +- gitea-38 +- chroma +- coherence +links: [] +modified: '2026-04-14T21:13:22+00:00' +summary: Architecture and implementation plan for refining Saga's concept taxonomy + and extraction schemas based on scientific research into cognitive categorization, + ontology design, and LLM extraction best practices. Covers MECE compliance, type + descriptions, schema structure, and validation approach. +tags: +- spec +- saga +- ingestion +- taxonomy +- redesign-plan +- gitea-37 +- gitea-38 +title: Saga Concept Taxonomy & Extraction Schema — Redesign Plan +--- + +# Saga Concept Taxonomy & Extraction Schema — Redesign Plan + +**Author:** Atlas (Librarian & Historian) +**Date:** 2025-07-12 +**Status:** Proposed — awaiting developer review +**Tracks:** Gitea #37 (taxonomy refinement), Gitea #38 (collection architecture redesign) +**Research basis:** Hermione `rs_concept_taxonomy_science` — 51 extracted concepts across 4 chapters + +--- + +## 1. Purpose & Scope + +This plan provides a grounded, phased approach for tuning Saga's concept taxonomy and extraction schemas from their current working state to a state backed by established research in cognitive science, ontology engineering, and LLM extraction design. + +**What this covers:** +- Concept type definitions (the "what we extract" layer) +- Concept type descriptions (the "how to recognize it" layer) +- Schema structure and type ordering (the "how we organize it" layer) +- Validation and quality measurement (the "how we know it works" layer) + +**What this does NOT cover:** +- Pipeline architecture changes (parsing, chunking, summarization — separate work) +- Coherence graph redesign (related but distinct) +- Chroma collection restructuring (Gitea #38 — this plan feeds into it) + +**Design philosophy:** We are not rebuilding — we are tuning. The current 7-source-type, 7-schema architecture is structurally sound. The research confirms the direction and identifies specific seams that need tightening. + +--- + +## 2. Research Summary — What the Science Says + +The following principles are drawn from Hermione's research session, grounded in primary sources. Each principle is stated with its source and mapped to a specific implication for Saga. + +### 2.1 Cognitive Science Foundations (Rosch 1978, Gozli 2023) + +**Principle — Basic-level categories:** Human categorization operates at a "basic level" that maximizes both cue validity (how well features predict category membership) and category resemblance (how similar members are to each other). This is the level at which people (and by extension, LLMs) classify most reliably and quickly. + +> *"These principles result in basic-level categories that maximize both total cue validity and category resemblance."* — Rosch (1978) + +**Implication for Saga:** Our concept types should sit at this basic level — specific enough to be actionable, general enough for reliable classification. The test: can an LLM extractor reliably assign a concept to one type without hesitation? If extraction accuracy shows high ambiguity between adjacent types, we're either too fine (too many types) or too coarse (types that serve multiple purposes). + +**Principle — Prototypicality effects:** The degree to which an instance matches the prototype of its category directly affects processing speed, learning efficiency, and classification confidence. Atypical instances are harder to classify. + +> *"Effects of prototypicality on processing speed, learning, item output, and natural language use are documented."* — Rosch (1978) + +**Implication for Saga:** Each concept type should have clear prototypical examples in its description. The extractor prompt should include "this IS a claim / this is NOT a claim" examples to anchor classification. + +**Principle — Sense vs. choice categories (Mammen/Gozli):** Some categories operate by logic of types (interchangeable instances — "sense"), while others operate by logic of individuals (unique, irreplaceable instances — "choice"). Deviations from basic-level toward choice categories reveal subjectivity. + +**Implication for Saga:** Our concept types are "sense" categories (the claim in chapter 3 is interchangeable with the claim in chapter 7 at the type level). This is correct. We should not try to make types that encode the specific instance's uniqueness — that's what the `verbatim` text and metadata are for. + +### 2.2 Taxonomy & Ontology Design Best Practices + +**Principle — MECE is non-negotiable:** Categories must be Mutually Exclusive (no concept should fit two types) and Collectively Exhaustive (every extractable concept should fit at least one type). + +> *"Key principles: MECE categories"* — NNGroup / Taxon.dev (multiple sources) +> *"choose a simple intuitive structure — MECE categories (mutually exclusive, collectively exhaustive)"* — Taxon.dev + +**Implication for Saga:** This is the single most important audit criterion. Current areas of concern (detailed in §3.1): + +- `claim` vs. `contradiction` (book schema) — a contradiction is semantically a claim that targets another view +- `principle` vs. `framework` (book schema) — a framework often is a set of principles +- `claim` vs. `evidence` (article schema) — a fact that serves as evidence is also a claim about reality + +**Principle — Start with a clear objective:** Taxonomy design must begin by defining what problem the system solves. + +> *"start with a clear objective — define what problem you're solving"* — Taxon.dev + +**Implication for Saga:** Our objective is clear but should be explicitly stated in the config: *"Enable retrieval of source-grounded knowledge atoms by type, with sufficient precision that an agent can filter by concept_type and get results matching their intent."* + +**Principle — Consistent, meaningful labels:** Taxonomy terms must use standardized terminology from the relevant domain. + +> *"use consistent meaningful labels — standardized terminology from the domain"* — NNGroup + +**Implication for Saga:** Our type names are already domain-standard for the most part (`hypothesis`, `finding`, `method` from academic research; `obligation`, `right`, `exception` from legal analysis). The only questionable label is `contradiction` — in argumentation theory, the standard term is `counterargument` or `rebuttal`, which we already use in the article schema. Consider aligning. + +**Principle — Iterative testing and refinement:** Taxonomies must be validated against real data and refined. + +> *"test and refine — evaluate in real-world scenarios"* — Taxon.dev + +**Implication for Saga:** We need a feedback loop — extract from a known document, classify by hand, compare to extractor output, measure agreement, adjust types. This is detailed in §5. + +**Principle — 10-step taxonomy building process:** Content inventory → check existing standards → identify key concepts → define hierarchy → apply MECE → test with real content → label consistently → document governance → plan maintenance → deploy and monitor. + +**Implication for Saga:** We've done steps 1-3 implicitly. Steps 4-6 are what this plan addresses. Steps 7-10 are the validation and governance covered in §5. + +### 2.3 Ontology Design Schools (Semantic Arts) + +Eight schools of ontology design were identified. The two most relevant to Saga: + +**Data-Centric School:** Modest class counts (<500), implemented and populated, high-level domain descriptions. Equivalent class restrictions enable inference. *This is closest to our current approach — a working, populated taxonomy with a small number of well-defined types.* + +**NLP/LLM School:** Machine-generated ontologies with hallucination risk. Best practice: controlled document sets + human in the loop. *This is literally what we're doing — an LLM extracts concepts from documents, and the results go through Atlas for review.* + +**Principle — Modular design:** Complex systems should be broken into smaller, manageable components. + +> *"use modular design — break into smaller manageable modules"* — Semantic Arts + +**Implication for Saga:** Our current 7-schema architecture is already modular (one schema per source type). This should be preserved. The module boundary should remain at the source-type level. + +**Principle — Define scope and purpose explicitly.** + +> *"define scope and purpose"* — Semantic Arts + +**Implication for Saga:** Each schema should carry an explicit scope statement describing what it's optimized to extract. Currently the schema name (`book`, `academic_paper`) implies the scope, but an explicit purpose statement would improve extractor performance. + +### 2.4 Argumentation Theory (Visser et al. 2021) + +**Principle — Classification difficulty is real:** Even trained annotators achieved only moderate inter-annotator agreement when classifying arguments into established typologies (Walton's 60 schemes, Wagemans' 36 types). Many arguments resisted classification entirely. + +> *"Key insight: argument scheme annotation is difficult — inter-annotator agreement is moderate"* — Visser et al. (2021) + +**Implication for Saga:** This is a humility check. We should expect imperfect classification and design for it rather than trying to eliminate it. Specific mitigations: + +- Keep type counts low (6-7 per schema is the right range — don't expand to 10+) +- Include a `confidence` field (already present) and use it downstream +- Include an `unclassifiable` or `other` fallback type, OR accept that some concepts will be imperfectly classified and rely on semantic search rather than type filtering for those cases +- The recommendation is to NOT add an `other` type — it becomes a junk drawer. Instead, accept the imperfection and document it. + +**Principle — Decision tree heuristics help:** Providing annotators (or LLMs) with a binary decision tree improves classification consistency. + +> *"A decision tree heuristic helps annotators classify arguments through binary questions."* — Visser et al. (2021) + +**Implication for Saga:** The extractor prompt should include a decision tree or prioritized set of disambiguation questions. Example: "Is this a general rule the author advocates? → `principle`. Is it a structured multi-part model? → `framework`. If you can't distinguish, default to `principle`." + +### 2.5 Extraction Schema Design for LLMs (LandingAI 2025) + +**Principle — Descriptive field names with contextual descriptions:** Field names must clearly indicate what to extract. Descriptions should specify format, currency, length constraints, and examples. + +> *"field names should clearly indicate what data to extract; descriptions provide additional context that improves accuracy — specify time formats, currency, expected length"* — LandingAI (2025) + +**Implication for Saga:** Our concept type descriptions are currently short labels ("A stated argument or position the author defends"). They should be expanded to include: +- **What it looks like** (linguistic cues: "typically uses declarative sentences, may include 'I argue', 'we show', 'the evidence suggests'") +- **What it is NOT** (disambiguation: "not a definition of a term, not a general observation") +- **Examples** (1-2 prototypical instances) + +**Principle — Mirror document structure:** Schema should reflect how data appears in the source document. + +> *"organize schema to reflect how data appears in parsed Markdown and how it will be used; use nested objects for hierarchical data; match document flow order"* — LandingAI (2025) + +**Implication for Saga:** Within each schema, types should be ordered by likelihood of occurrence in that source type. For example, in `academic_paper`, `finding` should come first (it's the most common and important), not `hypothesis`. In `book`, `claim` first is correct. In `fiction`, `theme` first is correct. + +**Principle — Strict data type specification:** Prevent downstream conversion errors by specifying correct types. + +> *"use correct types (string, number, integer, boolean, array, object) to prevent downstream conversion errors"* — LandingAI (2025) + +**Implication for Saga:** Review the concept metadata schema in `config.yaml` to ensure all fields have strict type declarations. The `confidence` field should be a float, `verbatim` a boolean, `terms` an array of strings, etc. + +--- + +## 3. Current State Analysis — Where the Seams Are + +### 3.1 MECE Audit of Each Schema + +#### `book` schema — issues found + +| Problem | Types | Why it's not MECE | Recommended resolution | +|---|---|---|---| +| `claim` and `contradiction` overlap | claim, contradiction | A contradiction is a claim that targets another view. The extractor must decide: is this "the author's argument" or "the author's challenge to another argument"? Both are claims structurally. | **Rename `contradiction` to `counterargument`** (aligns with argument schema terminology used in `article`). Add to description: "A claim that explicitly targets and opposes another stated position. If the author simply states an unconventional view without targeting a specific alternative, classify as `claim`." This creates a cleaner boundary: `counterargument` requires a target; `claim` does not. | +| `principle` and `framework` overlap | principle, framework | A framework (e.g., "Dual Process Theory") often contains principles (e.g., "System 1 is fast and automatic"). The boundary between "a general rule" and "a structured model of thinking" depends on scale, which is subjective. | **Keep both but sharpen descriptions.** `principle` = "a single, standalone rule or guideline." `framework` = "a named, multi-component system or model that organizes multiple ideas into a structured whole." Key test: can you name it? Does it have parts? If yes → framework. If it's one rule → principle. | +| `question` is a weak type | question | Open questions are rarely the primary knowledge atom a reader is searching for. They're more like metadata about what the author is investigating. | **Demote but keep.** Move `question` to the end of the type list (lowest priority). Consider whether it should be extracted at all, or whether it's better captured in section summaries. **Recommendation: keep for now, monitor extraction frequency.** | + +#### `academic_paper` schema — issues found + +| Problem | Types | Why it's not MECE | Recommended resolution | +|---|---|---|---| +| `hypothesis` and `finding` can overlap | hypothesis, finding | A confirmed hypothesis is reported as a finding. The same statement can be both depending on where in the paper it appears (intro = hypothesis, results = finding). | **Disambiguate by position AND framing.** `hypothesis` = "a prediction or proposition being tested, stated before results." `finding` = "an observed result or outcome, stated after methodology." If the extractor can't determine position, default to `finding`. | +| `contribution` is orphaned | contribution | Contributions are often a mix of findings, methods, and frameworks. They rarely appear as standalone knowledge atoms. | **Keep but narrow.** `contribution` = "a specific claimed novelty or advance that the authors explicitly identify as their contribution, typically in the introduction or discussion." | + +#### `article` schema — issues found + +| Problem | Types | Why it's not MECE | Recommended resolution | +|---|---|---|---| +| `claim` and `evidence` overlap | claim, evidence | A cited statistic ("60% of voters...") is evidence for a claim, but it's also a claim about reality. | **Disambiguate by role in the argument.** `claim` = "the author's own assertion or thesis." `evidence` = "a fact, statistic, or external source cited to support a claim." Key test: did the author produce this, or did they cite it? Author-produced → claim. Cited from elsewhere → evidence. | + +#### `technical_document` schema — no issues + +This schema is already well-MECE. The types (`definition`, `requirement`, `procedure`, `parameter`, `warning`, `example`) have clear structural boundaries. **No changes recommended.** + +#### `instruction_manual` schema — no issues + +Similarly clean. `step`, `prerequisite`, `warning`, `tip`, `troubleshooting`, `example` are well-separated. **No changes recommended.** + +#### `legal_document` schema — no issues + +`obligation`, `right`, `condition`, `exception`, `definition`, `penalty` are standard legal ontology terms with clear boundaries. **No changes recommended.** + +#### `fiction` schema — minor issue + +| Problem | Types | Why it's not MECE | Recommended resolution | +|---|---|---|---| +| `relationship` and `character_trait` can overlap | relationship, character_trait | A dynamic between characters can reveal character traits, and vice versa. | **Minor — acceptable as-is.** The boundary is: `relationship` describes the interaction between two+ characters. `character_trait` describes a quality of a single character. This is sufficiently distinct. | + +### 3.2 Type Ordering Audit + +Current default types (the type the extractor falls back to) are generally correct: + +| Schema | Current default | Assessment | +|---|---|---| +| `book` | `claim` | ✓ Correct — claims are the primary knowledge atoms | +| `academic_paper` | `finding` | ✓ Correct — findings are what researchers search for | +| `article` | `claim` | ✓ Correct | +| `technical_document` | `definition` | ✓ Correct — terms and concepts are the primary referents | +| `instruction_manual` | `tip` | ⚠️ Questionable — `step` may be more commonly extracted and useful | +| `legal_document` | `obligation` | ✓ Correct | +| `fiction` | `theme` | ✓ Correct | + +**Recommendation:** Change `instruction_manual` default from `tip` to `step`. + +### 3.3 Description Quality Audit + +Current descriptions are single-sentence labels. Per LandingAI's research, they should be expanded to include linguistic cues, disambiguation rules, and prototypical examples. All 7 schemas need this treatment. See §4.1 for the specification. + +--- + +## 4. Architecture Design — What to Build + +### 4.1 Enhanced Concept Type Specification Format + +Each concept type in `config.yaml` should be upgraded from a single-line description to a structured specification: + +```yaml +types: + claim: + label: "Claim" + short_description: "The author's own assertion, thesis, or position" + long_description: | + A declarative statement that the author argues for, defends, or puts forward as true. + This is the author's voice making a case — not a cited fact, not a definition of a term, + and not a question. + linguistic_cues: + - "The author argues that..." + - "We believe..." + - "The evidence shows..." + - "It is clear that..." + disambiguation: + - "NOT a definition (use 'definition' for term explanations)" + - "NOT a cited external fact (use 'evidence' in article schema)" + - "NOT a general observation without argumentative force" + prototypical_examples: + - "System 1 operates automatically and quickly with little or no effort." + - "Expert intuition is more reliable than we think in structured environments." + priority: 1 # order within the schema prompt +``` + +**Fields explained:** + +| Field | Purpose | Required | +|---|---|---| +| `label` | Human-readable display name | Yes | +| `short_description` | One-line summary for compact contexts | Yes | +| `long_description` | Full description for the extractor prompt | Yes | +| `linguistic_cues` | Common phrasing patterns that signal this type | Recommended | +| `disambiguation` | Rules for what this is NOT, especially vs. adjacent types | Recommended | +| `prototypical_examples` | 2-3 canonical examples from real source material | Yes | +| `priority` | Extraction priority (lower = extracted first, ties broken by this) | Yes | + +### 4.2 Schema Scope Statements + +Each schema should include a scope statement that the classifier uses to select it: + +```yaml +schemas: + book: + source_types: ["book"] + scope: | + Optimized for long-form non-fiction where the author develops arguments, + frameworks, and principles across chapters. Prioritizes claims and + structured models over individual facts. + default_type: "claim" + types: { ... } +``` + +### 4.3 Decision Tree for Disambiguation + +For schemas with adjacent-type ambiguity (specifically `book`), the extractor prompt should include a decision tree. Example for `book` schema: + +``` +1. Is the author defining a term or concept precisely? → DEFINITION +2. Is the author presenting a named, multi-component model? → FRAMEWORK +3. Is the author stating a general rule or guideline? → PRINCIPLE +4. Is the author challenging a specific alternative view? → COUNTERARGUMENT +5. Is the author making any other assertion or argument? → CLAIM +6. Is the author illustrating a point with a concrete case? → EXAMPLE +7. Is the author raising something they don't answer? → QUESTION +``` + +This tree should be generated from the type priorities and embedded in the extraction prompt. It gives the LLM a deterministic fallback path when classification is ambiguous — directly implementing Visser et al.'s finding that decision trees improve inter-annotator agreement. + +### 4.4 Concept Metadata Schema Additions + +Add to the concept metadata in `config.yaml`: + +```yaml +concept_metadata: + # Existing fields preserved... + + # New fields: + extraction_rationale: # Short explanation of why this type was chosen (string, optional) + adjacent_type_score: # If the extractor considered a second type, record it (string, optional) + source_position: # Where in the document structure this appeared (string: "introduction", "body", "conclusion", "methodology", "results") +``` + +**Why:** `extraction_rationale` and `adjacent_type_score` give us the data we need to measure classification ambiguity without re-reading the source. `source_position` enables position-based disambiguation (e.g., distinguishing hypothesis from finding in academic papers). + +--- + +## 5. Validation & Quality Measurement Plan + +### 5.1 Calibration Dataset + +Create a gold-standard calibration dataset: + +1. **Select 3-5 representative documents** (one per major schema: book, academic_paper, article, plus one wild card) +2. **Hand-annotate 50-100 concepts per document** with type assignments +3. **Record disambiguation notes** (why this was classified as X and not Y) +4. **Store in `validation/calibration/`** as JSON files with schema: + ```json + { + "source": "Thinking, Fast and Slow", + "section": "Ch. 1, Sec. 2", + "verbatim": "System 1 operates automatically...", + "hand_label": "claim", + "rationale": "Author's direct assertion, not a definition", + "adjacent_considered": "definition", + "notes": "Could be confused with definition because it describes System 1" + } + ``` + +### 5.2 Inter-Rater Reliability Metric + +After running the extractor against the calibration dataset: + +- **Measure per-type precision and recall** against hand labels +- **Identify the confusion matrix** (which types are most commonly confused with which) +- **Target: ≥85% agreement** on primary type assignment +- **Acceptable: ≥70%** with documented confusion pairs +- **Below 70% on any type pair:** triggers a merge or re-description of those types + +### 5.3 Feedback Loop + +``` +[Calibration dataset] → [Run extractor] → [Compare to hand labels] + → [Confusion matrix] → [Adjust type descriptions] + → [Re-run] → [Measure improvement] + → [When stable: promote descriptions to config.yaml] +``` + +This is not a one-time exercise. As new source types are added or existing schemas are used on new document types, the calibration dataset should grow. + +### 5.4 Governance + +- **Type changes require:** Updated description in `config.yaml` + re-run against calibration dataset + update to `concept-taxonomy-reference.md` +- **New type additions require:** Justification against MECE principle (why existing types don't cover it) + calibration dataset entries +- **Type removals require:** Evidence from extraction logs that the type is used <2% of the time or is never the primary classification +- **Review cadence:** After every 5 new ingestion jobs, review type distribution and confusion patterns + +--- + +## 6. Implementation Phases + +### Phase 1 — Foundation (config.yaml changes) + +**Goal:** Upgrade the type specification format without changing any type names or boundaries yet. + +**Tasks:** +1. [ ] Add `scope` field to each of the 7 schemas in `config.yaml` +2. [ ] Add `long_description`, `linguistic_cues`, `disambiguation`, `prototypical_examples`, `priority` to each concept type in each schema +3. [ ] Rename `contradiction` → `counterargument` in `book` schema +4. [ ] Change `instruction_manual` default type from `tip` to `step` +5. [ ] Add new metadata fields: `extraction_rationale`, `adjacent_type_score`, `source_position` +6. [ ] Update extractor prompt template to include decision tree for schemas with adjacent-type ambiguity +7. [ ] Update `docs/specs/concept-taxonomy-reference.md` to reflect all changes + +**Deliverable:** Updated `config.yaml` with enhanced type specs, updated reference doc. + +**Estimated effort:** Small — mostly YAML editing and prompt engineering. + +### Phase 2 — Calibration (validation infrastructure) + +**Goal:** Build the measurement infrastructure to validate that Phase 1 changes improve extraction quality. + +**Tasks:** +1. [ ] Create `validation/calibration/` directory structure +2. [ ] Hand-annotate calibration dataset (3-5 documents, 50-100 concepts each) +3. [ ] Build comparison script: `scripts/validate_extraction.py` that takes extractor output and calibration data, produces confusion matrix and per-type metrics +4. [ ] Run baseline measurement with current extractor against calibration dataset (before Phase 1 changes, if possible — otherwise against Phase 1 output) +5. [ ] Document baseline metrics in `validation/baseline_metrics.md` + +**Deliverable:** Calibration dataset, validation script, baseline metrics. + +**Estimated effort:** Medium — annotation is the bottleneck. + +### Phase 3 — Iterate (tuning cycle) + +**Goal:** Use calibration data to tune type descriptions until ≥85% agreement is achieved. + +**Tasks:** +1. [ ] Run extractor with Phase 1 type specs against calibration dataset +2. [ ] Generate confusion matrix +3. [ ] For each confused pair, either: (a) sharpen descriptions with more specific disambiguation rules, or (b) if confusion persists across iterations, consider merging the types +4. [ ] Re-run and re-measure +5. [ ] When stable, finalize descriptions in `config.yaml` +6. [ ] Re-index any existing ingested concepts that were extracted with the old type specs (optional — may not be worth it if changes are description-only) + +**Deliverable:** Finalized type specs with measured quality metrics. + +**Estimated effort:** Medium — depends on how many iterations are needed. + +### Phase 4 — Extend (optional, post-validation) + +**Goal:** Apply validated patterns to new source types or schema extensions. + +**Tasks:** +1. [ ] If adding new source types, use the enhanced type specification format from Phase 1 +2. [ ] Add calibration entries for new source types before deploying +3. [ ] Consider adding `source_position`-based extraction rules (e.g., in `academic_paper`, weight `hypothesis` higher in introduction sections, `finding` higher in results sections) + +**Deliverable:** Extension-ready architecture. + +--- + +## 7. Open Questions + +These are decisions that should be made before or during Phase 1, but don't block starting the work: + +1. **Should we add `source_position` to the extraction pipeline?** This requires the extractor to know which section of the document it's processing (introduction vs. body vs. conclusion). Currently, concepts are extracted per-section, and section context is in the prompt, but it's not explicitly labeled as a structural position. Adding it would help with hypothesis/finding disambiguation in academic papers. + +2. **Should `extraction_rationale` be stored in Chroma metadata or only in the extraction JSON?** Storing it in Chroma adds useful debugging data but increases storage per concept. Recommendation: store in extraction JSON, not in Chroma. + +3. **Should we re-index existing ingested concepts after type changes?** If we rename `contradiction` → `counterargument`, existing concepts tagged `contradiction` will be orphaned from the new taxonomy. Options: (a) bulk update the type label in Chroma, (b) leave them as-is and accept the inconsistency, (c) re-extract from source. Recommendation: (a) bulk update — it's a simple rename. + +4. **What's the threshold for creating a new schema vs. extending an existing one?** If we ingest a new document type (e.g., `podcast_transcript`), should it get its own schema or use the closest existing one? Recommendation: start by mapping to the closest existing schema, and only create a new schema if calibration shows ≥30% of concepts don't fit any existing type cleanly. + +--- + +## 8. What This Plan Does NOT Change + +To be explicit about stability: + +- **7 source types** — keeping all 7 (book, academic_paper, article, technical_document, instruction_manual, legal_document, fiction) +- **7 schemas** — keeping the one-schema-per-source-type architecture +- **7 coherence relationship types** — not touched by this plan +- **Pipeline architecture** — parse → structure → summarize → classify → extract → index → coherence is preserved +- **Chroma collections** — `ingested_concepts`, `knowledge_base`, `agent_memory` structure preserved (Gitea #38 may address this separately) +- **Type counts** — staying at 6-7 types per schema (one rename, no additions or removals in Phase 1) + +--- + +## 9. References + +- Rosch, E. (1978). *Principles of Categorization.* — Basic-level categories, prototypicality, cognitive economy +- Gozli, D. (2023). *On the Relative Nature of Categories.* — Sense vs. choice categories, deviations from basic-level +- NNGroup / Taxon.dev — MECE taxonomy design, 10-step process, consistent labels +- Semantic Arts — 8 ontology design schools, modular design, scope definition +- Visser, S. et al. (2021). *Argument Scheme Annotation.* — Inter-annotator agreement, Walton's 60 schemes, Wagemans' Periodic Table, decision tree heuristics +- LandingAI (2025). *LLM Extraction Schema Design.* — Descriptive field names, document-mirroring, data type specificity +- Mammen, J. — Sense vs. choice categories (as discussed in Gozli 2023) +- Wagemans, J. — Periodic Table of Arguments (36 types, factorial typology) + +--- + +## Appendix A: Quick-Reference — Changes Summary + +| What | Current | Proposed | Phase | +|---|---|---|---| +| `book.contradiction` | `contradiction` | Rename to `counterargument` | 1 | +| `instruction_manual` default | `tip` | Change to `step` | 1 | +| Type descriptions | 1-sentence label | Structured spec (long desc + cues + disambiguation + examples + priority) | 1 | +| Schema scope | Implicit (schema name) | Explicit `scope` field | 1 | +| Extractor prompt | Type list only | Type list + decision tree for ambiguous schemas | 1 | +| Concept metadata | Current fields | Add `extraction_rationale`, `adjacent_type_score`, `source_position` | 1 | +| Validation infrastructure | None | Calibration dataset + comparison script + baseline metrics | 2 | +| Quality target | Not measured | ≥85% type agreement vs. hand labels | 3 | + +## Appendix B: Example Enhanced Type Spec — `book.claim` + +```yaml +claim: + label: "Claim" + short_description: "The author's own assertion, thesis, or position" + long_description: | + A declarative statement that the author argues for, defends, or puts forward as true. + This is the author's voice making a substantive case. Claims are the primary knowledge + atoms in non-fiction — they are what a reader would quote or reference when discussing + the book's arguments. + linguistic_cues: + - "I argue that..." + - "The evidence shows..." + - "It is clear that..." + - "We will see that..." + - "The key insight is..." + - Declarative sentences in the author's voice that assert something as true + disambiguation: + - "NOT a definition of a term (use 'definition' — definitions explain what something means, claims assert that something is true)" + - "NOT a cited external fact (use 'evidence' — claims are the author's own, evidence is borrowed)" + - "NOT a challenge to a specific alternative (use 'counterargument' — counterarguments explicitly target another view)" + - "NOT a general rule without argumentative force (use 'principle' — principles are prescriptive, claims are descriptive)" + prototypical_examples: + - "System 1 operates automatically and quickly with little or no effort and no sense of voluntary control." + - "The planning fallacy describes a tendency for people to underestimate the time, costs, and risks of future actions." + - "Expert intuition is more reliable than we think, but only in environments with regular feedback." + priority: 1 +``` \ No newline at end of file