VuongKB/docs/specs/saga-taxonomy-redesign-plan.md at main

Files

T

Atlas 3262a72a7a Update: docs/specs/saga-taxonomy-redesign-plan.md — Expanded scope from tuning pass to full redesign: added Phase 0 baseline measurement, section-aware extraction (§4.6), coherence layer redesign (§4.7), reordered phases, updated appendices, and revised §1/§8 to reflect the broader mandate.

2026-04-14 14:35:04 -07:00

57 KiB

Raw Permalink Blame History

created, id, keywords, links, modified, summary, tags, title

created

keywords

links

modified

summary

1. Purpose & Scope

This plan provides a grounded, phased approach for improving Saga's concept extraction quality — from the type definitions that guide extraction, through the prompts the LLM sees, to the coherence relationships that connect extracted concepts across documents.

The original plan was scoped as a tuning pass. After analysis, the scope has been expanded to include three additional high-ROI areas that the research supports and that are cheap to do now (4 completed jobs, small data footprint) but expensive to retrofit later.

What this covers:

Baseline measurement — knowing where we stand before changing anything
Concept type definitions — the "what we extract" layer
Concept type descriptions — the "how to recognize it" layer
Prompt assembly — how type specs become LLM instructions
Section-aware extraction — using document structure the pipeline already produces
Schema structure and type ordering — the "how we organize it" layer
Coherence relationship redesign — the "how concepts relate" layer
Validation and quality measurement — the "how we know it works" layer

What this does NOT cover:

Pipeline architecture changes (parsing, chunking, summarization — separate work)
Chroma collection restructuring (Gitea #38 — this plan feeds into it)
New source type or schema additions (future work, governed by §5.4)

Design philosophy: The pipeline architecture — parse → structure → summarize → classify → extract → index → coherence — is structurally sound. The 7-source-type, 7-schema modular design is exactly what the research recommends. We are not rebuilding the pipeline. We are rebuilding the prompts, the metadata, and the relationship vocabulary that the pipeline uses.

2. Research Summary — What the Science Says

The following principles are drawn from Hermione's research session, grounded in primary sources. Each principle is stated with its source and mapped to a specific implication for Saga.

2.1 Cognitive Science Foundations (Rosch 1978, Gozli 2023)

Principle — Basic-level categories: Human categorization operates at a "basic level" that maximizes both cue validity (how well features predict category membership) and category resemblance (how similar members are to each other). This is the level at which people (and by extension, LLMs) classify most reliably and quickly.

"These principles result in basic-level categories that maximize both total cue validity and category resemblance." — Rosch (1978)

Implication for Saga: Our concept types should sit at this basic level — specific enough to be actionable, general enough for reliable classification. The test: can an LLM extractor reliably assign a concept to one type without hesitation? If extraction accuracy shows high ambiguity between adjacent types, we're either too fine (too many types) or too coarse (types that serve multiple purposes).

Principle — Prototypicality effects: The degree to which an instance matches the prototype of its category directly affects processing speed, learning efficiency, and classification confidence. Atypical instances are harder to classify.

"Effects of prototypicality on processing speed, learning, item output, and natural language use are documented." — Rosch (1978)

Implication for Saga: Each concept type should have clear prototypical examples in its description. The extractor prompt should include "this IS a claim / this is NOT a claim" examples to anchor classification.

Principle — Sense vs. choice categories (Mammen/Gozli): Some categories operate by logic of types (interchangeable instances — "sense"), while others operate by logic of individuals (unique, irreplaceable instances — "choice"). Deviations from basic-level toward choice categories reveal subjectivity.

Implication for Saga: Our concept types are "sense" categories (the claim in chapter 3 is interchangeable with the claim in chapter 7 at the type level). This is correct. We should not try to make types that encode the specific instance's uniqueness — that's what the verbatim text and metadata are for.

2.2 Taxonomy & Ontology Design Best Practices

Principle — MECE is non-negotiable: Categories must be Mutually Exclusive (no concept should fit two types) and Collectively Exhaustive (every extractable concept should fit at least one type).

"Key principles: MECE categories" — NNGroup / Taxon.dev (multiple sources) "choose a simple intuitive structure — MECE categories (mutually exclusive, collectively exhaustive)" — Taxon.dev

Implication for Saga: This is the single most important audit criterion. Current areas of concern (detailed in §3.1):

claim vs. contradiction (book schema) — a contradiction is semantically a claim that targets another view
principle vs. framework (book schema) — a framework often is a set of principles
claim vs. evidence (article schema) — a fact that serves as evidence is also a claim about reality

Principle — Start with a clear objective: Taxonomy design must begin by defining what problem the system solves.

"start with a clear objective — define what problem you're solving" — Taxon.dev

Implication for Saga: Our objective is clear but should be explicitly stated in the config: "Enable retrieval of source-grounded knowledge atoms by type, with sufficient precision that an agent can filter by concept_type and get results matching their intent."

Principle — Consistent, meaningful labels: Taxonomy terms must use standardized terminology from the relevant domain.

"use consistent meaningful labels — standardized terminology from the domain" — NNGroup

Implication for Saga: Our type names are already domain-standard for the most part (hypothesis, finding, method from academic research; obligation, right, exception from legal analysis). The only questionable label is contradiction — in argumentation theory, the standard term is counterargument or rebuttal, which we already use in the article schema. Consider aligning.

Principle — Iterative testing and refinement: Taxonomies must be validated against real data and refined.

"test and refine — evaluate in real-world scenarios" — Taxon.dev

Implication for Saga: We need a feedback loop — extract from a known document, classify by hand, compare to extractor output, measure agreement, adjust types. This is detailed in §5.

Principle — 10-step taxonomy building process: Content inventory → check existing standards → identify key concepts → define hierarchy → apply MECE → test with real content → label consistently → document governance → plan maintenance → deploy and monitor.

Implication for Saga: We've done steps 1-3 implicitly. Steps 4-6 are what this plan addresses. Steps 7-10 are the validation and governance covered in §5.

2.3 Ontology Design Schools (Semantic Arts)

Eight schools of ontology design were identified. The two most relevant to Saga:

Data-Centric School: Modest class counts (<500), implemented and populated, high-level domain descriptions. Equivalent class restrictions enable inference. This is closest to our current approach — a working, populated taxonomy with a small number of well-defined types.

NLP/LLM School: Machine-generated ontologies with hallucination risk. Best practice: controlled document sets + human in the loop. This is literally what we're doing — an LLM extracts concepts from documents, and the results go through Atlas for review.

Principle — Modular design: Complex systems should be broken into smaller, manageable components.

"use modular design — break into smaller manageable modules" — Semantic Arts

Implication for Saga: Our current 7-schema architecture is already modular (one schema per source type). This should be preserved. The module boundary should remain at the source-type level.

Principle — Define scope and purpose explicitly.

"define scope and purpose" — Semantic Arts

Implication for Saga: Each schema should carry an explicit scope statement describing what it's optimized to extract. Currently the schema name (book, academic_paper) implies the scope, but an explicit purpose statement would improve extractor performance.

2.4 Argumentation Theory (Visser et al. 2021, Walton, Wagemans)

Principle — Classification difficulty is real: Even trained annotators achieved only moderate inter-annotator agreement when classifying arguments into established typologies (Walton's 60 schemes, Wagemans' 36 types). Many arguments resisted classification entirely.

"Key insight: argument scheme annotation is difficult — inter-annotator agreement is moderate" — Visser et al. (2021)

Implication for Saga: This is a humility check. We should expect imperfect classification and design for it rather than trying to eliminate it. Specific mitigations:

Keep type counts low (6-7 per schema is the right range — don't expand to 10+)
Include a confidence field (already present) and use it downstream
Include an unclassifiable or other fallback type, OR accept that some concepts will be imperfectly classified and rely on semantic search rather than type filtering for those cases
The recommendation is to NOT add an other type — it becomes a junk drawer. Instead, accept the imperfection and document it.

Principle — Decision tree heuristics help: Providing annotators (or LLMs) with a binary decision tree improves classification consistency.

"A decision tree heuristic helps annotators classify arguments through binary questions." — Visser et al. (2021)

Implication for Saga: The extractor prompt should include a binary decision tree where each fork asks a single yes/no question. The tree should front-load the most common types so the extractor reaches the right answer quickly. See §4.3 for the full specification.

Principle — Content vs. function distinction: In argumentation theory, what a proposition IS (its content) is distinct from what it DOES (its function in an argument). A statistic can be a claim (when the author asserts it) or evidence (when the author uses it to support something else). The same content plays different functional roles.

"The content vs function distinction is critical: what a proposition IS vs what it DOES in the argument." — Visser et al. (2021)

Implication for Saga: This distinction is exactly what our coherence relationship types should capture. Currently agrees and contradicts describe stance (positive vs. negative), but they don't capture function (supporting, warranting, rebutting, qualifying). The coherence layer should be redesigned to reflect argument function, not just sentiment alignment. See §4.7.

Principle — Walton's 60 schemes and Wagemans' 36 types: These are comprehensive typologies of argument schemes — the patterns of reasoning that connect premises to conclusions. They include categories like "argument from example," "argument from cause to effect," "practical reasoning," and "ad hominem variants."

"Walton's taxonomy: empirically-motivated classification with 60 upper-level argument schemes" — Visser et al. (2021) "Wagemans' Periodic Table of Arguments: a priori exhaustive factorial typology comprising 36 argument types" — Visser et al. (2021)

Implication for Saga: We should not adopt 60 or 36 relationship types — that would be absurd for our use case. But the dimensional structure of these typologies is instructive. Wagemans' factorial approach (statement type × assertion type × subject matter) suggests that our coherence relationships could be organized along a small number of orthogonal dimensions rather than as a flat list of 7 labels. See §4.7.

2.5 Extraction Schema Design for LLMs (LandingAI 2025)

Principle — Descriptive field names with contextual descriptions: Field names must clearly indicate what to extract. Descriptions should specify format, currency, length constraints, and examples.

"field names should clearly indicate what data to extract; descriptions provide additional context that improves accuracy — specify time formats, currency, expected length" — LandingAI (2025)

Implication for Saga: Our concept type descriptions are currently short labels ("A stated argument or position the author defends"). They should be expanded to include:

What it looks like (linguistic cues: "typically uses declarative sentences, may include 'I argue', 'we show', 'the evidence suggests'")
What it is NOT (disambiguation: "not a definition of a term, not a general observation")
Examples (1-2 prototypical instances)

Principle — Mirror document structure: Schema should reflect how data appears in the source document.

"organize schema to reflect how data appears in parsed Markdown and how it will be used; use nested objects for hierarchical data; match document flow order" — LandingAI (2025)

Implication for Saga: Two implications:

Within each schema, types should be ordered by likelihood of occurrence in that source type. For example, in academic_paper, finding should come first (it's the most common and important), not hypothesis. In book, claim first is correct.
The extraction prompt should receive section-level context — the chapter title, section heading, and position in the document — not just the raw text. The pipeline already produces this information during the structure and summarize phases. We should pass it through to extraction. See §4.6.

Principle — Strict data type specification: Prevent downstream conversion errors by specifying correct types.

"use correct types (string, number, integer, boolean, array, object) to prevent downstream conversion errors" — LandingAI (2025)

Implication for Saga: Review the concept metadata schema in config.yaml to ensure all fields have strict type declarations. The confidence field should be a float, verbatim a boolean, terms an array of strings, etc.

3. Current State Analysis — Where the Seams Are

3.1 MECE Audit of Each Schema

`book` schema — issues found

Problem	Types	Why it's not MECE	Recommended resolution
`claim` and `contradiction` overlap	claim, contradiction	A contradiction is a claim that targets another view. The extractor must decide: is this "the author's argument" or "the author's challenge to another argument"? Both are claims structurally.	Rename `contradiction` to `counterargument` (aligns with argument schema terminology used in `article`). Add to description: "A claim that explicitly targets and opposes another stated position. If the author simply states an unconventional view without targeting a specific alternative, classify as `claim`." This creates a cleaner boundary: `counterargument` requires a target; `claim` does not.
`claim` and `principle` overlap	claim, principle	Both are the author putting something forward. A claim asserts what is true; a principle prescribes what to do or how to think. The boundary can be fuzzy when a claim reads like a rule ("Practice deliberate feedback").	Sharpen the disambiguation rule. The test is: can you imagine someone following this? If yes → `principle`. If it's a statement about reality (even a strong one) → `claim`. "System 1 operates automatically" is a claim — it describes how things are. "Seek regular feedback in structured environments" is a principle — it tells you what to do.
`principle` and `framework` overlap	principle, framework	A framework (e.g., "Dual Process Theory") often contains principles (e.g., "System 1 is fast and automatic"). The boundary between "a general rule" and "a structured model of thinking" depends on scale, which is subjective.	Keep both but sharpen descriptions. `principle` = "a single, standalone rule or guideline." `framework` = "a named, multi-component system or model that organizes multiple ideas into a structured whole." Key test: can you name it? Does it have parts? If yes → framework. If it's one rule → principle.
`question` is a weak type	question	Open questions are rarely the primary knowledge atom a reader is searching for. They're more like metadata about what the author is investigating.	Demote but keep. Move `question` to the end of the type list (lowest priority). Consider whether it should be extracted at all, or whether it's better captured in section summaries. Recommendation: keep for now, monitor extraction frequency.

`academic_paper` schema — issues found

Problem	Types	Why it's not MECE	Recommended resolution
`hypothesis` and `finding` can overlap	hypothesis, finding	A confirmed hypothesis is reported as a finding. The same statement can be both depending on where in the paper it appears (intro = hypothesis, results = finding).	Disambiguate by tense and framing, not by position. `hypothesis` = "a prediction or proposition framed in future or conditional tense about what the study expects to find." `finding` = "an observed result framed in past tense about what the study found." Key test: is the author predicting what will happen (hypothesis) or reporting what did happen (finding)? Position in the document can be a corroborating signal if available (§4.6), but tense/framing is the primary disambiguation criterion and works without structural metadata.
`contribution` is orphaned	contribution	Contributions are often a mix of findings, methods, and frameworks. They rarely appear as standalone knowledge atoms.	Keep but narrow. `contribution` = "a specific claimed novelty or advance that the authors explicitly identify as their contribution, typically in the introduction or discussion."

`article` schema — issues found

Problem	Types	Why it's not MECE	Recommended resolution
`claim` and `evidence` overlap	claim, evidence	A cited statistic ("60% of voters...") is evidence for a claim, but it's also a claim about reality.	Disambiguate by role in the argument. `claim` = "the author's own assertion or thesis." `evidence` = "a fact, statistic, or external source cited to support a claim." Key test: did the author produce this, or did they cite it? Author-produced → claim. Cited from elsewhere → evidence.

`technical_document` schema — no issues

This schema is already well-MECE. The types (definition, requirement, procedure, parameter, warning, example) have clear structural boundaries. No changes recommended.

`instruction_manual` schema — no issues

Similarly clean. step, prerequisite, warning, tip, troubleshooting, example are well-separated. No changes recommended.

`legal_document` schema — no issues

obligation, right, condition, exception, definition, penalty are standard legal ontology terms with clear boundaries. No changes recommended.

`fiction` schema — minor issue

Problem	Types	Why it's not MECE	Recommended resolution
`relationship` and `character_trait` can overlap	relationship, character_trait	A dynamic between characters can reveal character traits, and vice versa.	Minor — acceptable as-is. The boundary is: `relationship` describes the interaction between two+ characters. `character_trait` describes a quality of a single character. This is sufficiently distinct.

3.2 Type Ordering Audit

Current default types (the type the extractor falls back to) are generally correct:

Schema	Current default	Assessment
`book`	`claim`	✓ Correct — claims are the primary knowledge atoms
`academic_paper`	`finding`	✓ Correct — findings are what researchers search for
`article`	`claim`	✓ Correct
`technical_document`	`definition`	✓ Correct — terms and concepts are the primary referents
`instruction_manual`	`tip`	⚠️ Questionable — `step` may be more commonly extracted and useful
`legal_document`	`obligation`	✓ Correct
`fiction`	`theme`	✓ Correct

Recommendation: Change instruction_manual default from tip to step.

3.3 Description Quality Audit

Current descriptions are single-sentence labels. Per LandingAI's research, they should be expanded to include linguistic cues, disambiguation rules, and prototypical examples. All 7 schemas need this treatment. See §4.1 for the specification.

3.4 Coherence Relationship Audit

The current 7 coherence relationship types were designed well intuitively, but they were not grounded in the argumentation theory research. Reviewing them against the research:

Relationship	Current description	Assessment
`same_idea`	Two concepts express the same underlying idea	✓ Clear, no ambiguity
`agrees`	One concept supports or corroborates another	⚠️ Underspecified — "supports" and "corroborates" are different functions. Corroboration means independent evidence arriving at the same conclusion. Support means one concept provides reasoning for another. These are distinct argumentative functions.
`contradicts`	One concept opposes or refutes another	⚠️ Underspecified — "opposes" and "refutes" are different. One can oppose something without refuting it (disagreement vs. disproof).
`extends`	One concept builds upon or elaborates another	✓ Clear
`specializes`	One concept is a specific case of a more general one	✓ Clear
`exemplifies`	One concept provides a concrete example of another	✓ Clear
`undermines`	One concept weakens the basis for another without directly contradicting it	✓ Clear — this is actually the most nuanced current type

Key finding: The positive alignment types (agrees) and negative alignment types (contradicts) conflate stance with function. Argumentation theory distinguishes between these dimensions. See §4.7 for the proposed redesign.

4. Architecture Design — What to Build

4.1 Enhanced Concept Type Specification Format

Each concept type in config.yaml should be upgraded from a single-line description to a structured specification:

types:
  claim:
    label: "Claim"
    short_description: "The author's own assertion, thesis, or position"
    long_description: |
      A declarative statement that the author argues for, defends, or puts forward as true.
      This is the author's voice making a case — not a cited fact, not a definition of a term,
      and not a question.
    linguistic_cues:
      - "The author argues that..."
      - "We believe..."
      - "The evidence shows..."
      - "It is clear that..."
    disambiguation:
      - "NOT a definition (use 'definition' for term explanations)"
      - "NOT a cited external fact (use 'evidence' in article schema)"
      - "NOT a general observation without argumentative force"
    prototypical_examples:
      - "System 1 operates automatically and quickly with little or no effort."
      - "Expert intuition is more reliable than we think in structured environments."
    priority: 1  # order within the schema prompt

Fields explained:

Field	Purpose	Required
`label`	Human-readable display name	Yes
`short_description`	One-line summary for compact contexts	Yes
`long_description`	Full description for the extractor prompt	Yes
`linguistic_cues`	Common phrasing patterns that signal this type	Recommended
`disambiguation`	Rules for what this is NOT, especially vs. adjacent types	Recommended
`prototypical_examples`	2-3 canonical examples from real source material	Yes
`priority`	Extraction priority (lower = extracted first, ties broken by this)	Yes

4.2 Schema Scope Statements

Each schema should include a scope statement that the classifier uses to select it:

schemas:
  book:
    source_types: ["book"]
    scope: |
      Optimized for long-form non-fiction where the author develops arguments,
      frameworks, and principles across chapters. Prioritizes claims and
      structured models over individual facts.
    default_type: "claim"
    types: { ... }

4.3 Decision Tree for Disambiguation

For schemas with adjacent-type ambiguity (specifically book), the extractor prompt should include a binary decision tree — a series of yes/no questions where each fork leads to either a type assignment or the next question. This directly implements Visser et al.'s finding that binary decision trees improve classification consistency.

The tree should front-load the most common types so the extractor reaches the right answer quickly.

Example for book schema:

Is the author asserting something as true or arguing a position?
  YES:
    Is there a specific alternative view they are explicitly opposing?
      YES → COUNTERARGUMENT
      NO → CLAIM
  NO:
    Is the author defining a term or concept precisely?
      YES → DEFINITION
      NO:
        Is the author presenting a named, multi-component model or system?
          YES → FRAMEWORK
          NO:
            Is the author stating a rule about what to do or what generally applies?
              YES → PRINCIPLE
              NO:
                Is the author illustrating a point with a concrete instance?
                  YES → EXAMPLE
                  NO:
                    Is the author raising something they don't answer?
                      YES → QUESTION
                      NO → CLAIM  (default fallback)

Implementation notes:

This tree is generated from the priority ordering of types in the schema.
The root question targets the highest-priority type's core distinction (asserting something as true → claim, the most common and important type).
Each fork is a single yes/no question — the extractor never has to choose among 7 options simultaneously.
The final fallback is always the schema's default_type.
For schemas without significant ambiguity (e.g., legal_document), a decision tree is unnecessary — the standard type list suffices.

4.4 Prompt Assembly

The enhanced type specifications must be assembled into the actual LLM extraction prompt. This section specifies how.

Which fields go into the prompt, and in what order:

For each concept type in the schema, the prompt should include:

label — bolded header
long_description — full paragraph
linguistic_cues — bullet list (if present)
disambiguation — bullet list (if present)
prototypical_examples — numbered list (1-2 examples, not all)

Fields NOT included in the prompt: short_description (for human consumption only), priority (used for ordering only).

Token budget guidance:

A typical concept type spec assembled as above runs ~150-250 tokens. With 6-7 types per schema, that's ~1,000-1,750 tokens for the type definitions section — well within a standard extraction prompt context. If schemas grow beyond 10 types, consider using short_description instead of long_description and dropping linguistic_cues to stay within budget.

Decision tree placement:

If a decision tree is used (§4.3), it appears after the type definitions and before the extraction instructions. The prompt structure should be:

[Schema scope statement]
[Section context — chapter title, section heading, position] (see §4.6)
[Type definitions — ordered by priority]
[Decision tree for disambiguation]
[Extraction instructions — format, output schema, constraints]

Example assembled prompt fragment for book.claim:

## Claim
A declarative statement that the author argues for, defends, or puts forward as true.
This is the author's voice making a substantive case. Claims are the primary knowledge
atoms in non-fiction — they are what a reader would quote or reference when discussing
the book's arguments.

Signals: "I argue that...", "The evidence shows...", "It is clear that...", declarative
sentences asserting something as true.

Not to be confused with: definitions (explaining what a term means), cited facts (evidence
from elsewhere), challenges to specific alternatives (counterarguments), or prescriptive
rules (principles).

Examples:
1. "System 1 operates automatically and quickly with little or no effort and no sense of
   voluntary control."
2. "Expert intuition is more reliable than we think, but only in environments with regular
   feedback."

4.5 Concept Metadata Schema Additions

Phase 1 — essential addition:

concept_metadata:
  # Existing fields preserved...
  
  # New field:
  source_position:         # Where in the document structure this appeared
                           # (string: "introduction", "body", "conclusion", "methodology", "results")

source_position enables position-aware disambiguation as a corroborating signal (e.g., in academic_paper, statements in a "results" section are more likely to be finding). It is optional in Phase 1 — populated when the section structure is available from the parsed document.

Phase 5 — optional diagnostic fields:

These are deferred to Phase 5 and included only if calibration reveals systematic confusion that needs debugging:

  extraction_rationale:    # Short explanation of why this type was chosen (string, optional)
  adjacent_type_score:     # If the extractor considered a second type, record it (string, optional)

Why these are deferred: extraction_rationale adds prompt overhead for every extracted concept, and LLMs are known to confabulate post-hoc justifications rather than accurately reporting their reasoning. adjacent_type_score asks the LLM to quantify its own uncertainty — something LLMs are notably poor at. Neither field is used by the validation plan (§5), which compares against hand labels. Their value is purely for post-hoc debugging of systematic confusion patterns. They should only be added if Phase 3-4 calibration reveals a specific problem they would help diagnose.

4.6 Section-Aware Extraction

The pipeline already produces rich structural context during the structure and summarize phases. Currently, the extractor receives the raw section text and the schema type list, but not the chapter title, section heading, or position-in-document metadata. The LandingAI research and common sense both say this matters: a paragraph in "Chapter 3: The Anatomy of Bias" should be extracted differently than one in "Chapter 18: About the Author."

What to pass into the extraction prompt:

section_context:
  document_title: "Thinking, Fast and Slow"
  chapter_title: "The Characters of the System"
  chapter_number: 1
  section_title: "An Autonomous System"
  section_position: "early"       # early/middle/late within the document
  document_position: 0.05         # fraction of total document (0.0 to 1.0)

How this is assembled into the prompt:

The section context block appears at the top of the extraction prompt, before the type definitions:

You are extracting concepts from:
  Document: "Thinking, Fast and Slow"
  Chapter 1: "The Characters of the System"
  Section: "An Autonomous System"
  Position: Early in the document (5% through)

[Schema scope statement]
[Type definitions — ordered by priority]
[Decision tree for disambiguation]
[Extraction instructions]

How this improves extraction:

Position priors: Early in a book, concepts are more likely to be definition (introducing terms) or framework (establishing the author's model). Late in a book, concepts are more likely to be claim (arguing specific positions) or example (illustrating established ideas). The extractor doesn't need explicit rules for this — just seeing the position is enough for the LLM to adjust its priors.
Section title cues: A section titled "Experimental Evidence" primes the extractor to look for finding-type concepts. A section titled "What This Means" primes for claim or principle. Again, no explicit rules needed — the LLM picks up on these cues naturally.
Academic paper disambiguation: For academic_paper schemas, the section context directly resolves the hypothesis/finding ambiguity. Statements in "Introduction" or "Hypotheses" sections are overwhelmingly hypothesis; statements in "Results" sections are overwhelmingly finding.

Implementation:

The structural information is already available in structure.json and the summary files. The change is:

The extractor prompt builder reads section metadata from the existing pipeline outputs
Injects it as a context block at the top of the prompt
No changes to parsing or chunking

4.7 Coherence Relationship Redesign

The current 7 coherence types (same_idea, agrees, contradicts, extends, specializes, exemplifies, undermines) capture useful relationships but conflate two independent dimensions: stance (positive vs. negative alignment) and function (what role the relationship plays in the argumentative structure).

The argumentation theory research (Visser et al., Walton, Wagemans) provides a clearer vocabulary. The key insight from Wagemans is the factorial approach: relationship types can be described as combinations of a small number of independent dimensions rather than as a flat list.

Proposed dimensional structure:

The coherence relationship between two concepts should be described along three dimensions:

Dimension	Values	What it captures
Alignment	`supportive`, `opposing`, `neutral`	Does concept B strengthen, weaken, or not affect concept A?
Function	`corroborates`, `warrants`, `exemplifies`, `specializes`, `extends`, `rebuts`, `undermines`, `qualifies`	What is the argumentative role of the relationship?
Scope	`same`, `broader`, `narrower`	Is B at the same, higher, or lower level of abstraction than A?

How this maps to the current 7 types:

Current type	New representation
`same_idea`	alignment=`supportive`, function=`corroborates`, scope=`same`
`agrees`	alignment=`supportive`, function=`corroborates` or `warrants`, scope=any
`contradicts`	alignment=`opposing`, function=`rebuts`, scope=`same`
`extends`	alignment=`supportive`, function=`extends`, scope=`broader`
`specializes`	alignment=`supportive`, function=`specializes`, scope=`narrower`
`exemplifies`	alignment=`supportive`, function=`exemplifies`, scope=`narrower`
`undermines`	alignment=`opposing`, function=`undermines`, scope=any

What this adds (the new relationships the current system can't express):

New function	Description	Example
`warrants`	B provides the reasoning or evidence that justifies A	"Regular feedback improves performance" warrants "Expert intuition is reliable in structured environments"
`rebuts`	B directly contradicts A's conclusion	"System 1 is always unreliable" rebuts "Expert intuition is reliable in structured environments"
`qualifies`	B adds a condition, limitation, or scope restriction to A	"Only in environments with regular feedback" qualifies "Expert intuition is reliable"

Implementation approach:

The coherence phase of the pipeline currently uses an LLM to classify relationships into the 7 flat types. The change is:

Replace the flat classification with dimensional classification. The prompt asks the LLM to classify along each dimension independently. This is easier for the LLM (each dimension is a simple choice) and produces richer output.

Derive a primary relationship label from the dimensional scores. For backward compatibility and human readability, derive a label from the combination:

if alignment == supportive and function == corroborates and scope == same → "same_idea"
if alignment == supportive and function == warrants → "supports"
if alignment == opposing and function == rebuts → "contradicts"
if alignment == opposing and function == undermines → "undermines"
if alignment == opposing and function == qualifies → "qualifies"
... etc.

Store both the dimensional scores and the derived label. The label is used for display and filtering. The dimensional scores are used for graph analysis and agent reasoning.
Migration of existing coherence data: Existing relationships in coherence/graph.db can be re-classified by running the new dimensional prompt against the stored concept pairs. This is a batch operation, not a manual migration.

Why this is worth doing now:

The coherence graph is the feature that makes the KB more than a bag of concepts — it's what enables agents to reason about how ideas connect across documents.
The current types were designed intuitively and not revisited. The research provides a principled vocabulary.
There are only 4 completed jobs' worth of coherence data to migrate — the cost is minimal.
Delaying this means more data to migrate later and more downstream code that depends on the current flat type list.

5. Validation & Quality Measurement Plan

5.1 Calibration Dataset

Create a gold-standard calibration dataset:

Select 3-5 representative documents — prefer documents that have already been ingested so we can compare existing extractions directly against the hand annotations. One per major schema: book, academic_paper, article, plus one wild card.
Hand-annotate 50-100 concepts per document with type assignments, using the following process:
- Primary annotator: Atlas (the Librarian), working from the source text and the enhanced type specifications.
- Annotation protocol: For each concept, record the hand label, rationale, and any adjacent type considered. Use the same JSON schema the extractor produces so comparison is direct.
- Inter-annotator check: After the primary annotation is complete, a second annotator (the user or another agent) independently labels a 20% subset (10-20 concepts per document). Measure human-human agreement on this subset. If human-human agreement is below 90%, the type definitions themselves may be ambiguous and need sharpening before measuring LLM accuracy. This addresses the meta-problem Visser et al. identified: if humans can't agree, the taxonomy is the problem, not the annotator.
Record disambiguation notes — for every concept where the annotator hesitated or considered an alternative type, record why. These notes are the raw material for future description improvements.

Store in validation/calibration/ as JSON files with schema:

{
  "source": "Thinking, Fast and Slow",
  "section": "Ch. 1, Sec. 2",
  "verbatim": "System 1 operates automatically...",
  "hand_label": "claim",
  "rationale": "Author's direct assertion about how cognition works",
  "adjacent_considered": "definition",
  "notes": "Could be confused with definition because it describes System 1, but it's asserting a factual claim, not explaining a term"
}

5.2 Inter-Rater Reliability Metric

After running the extractor against the calibration dataset:

Measure per-type precision and recall against hand labels
Identify the confusion matrix (which types are most commonly confused with which)
Target: ≥85% agreement on primary type assignment
Acceptable: ≥70% with documented confusion pairs
Below 70% on any type pair: triggers a merge or re-description of those types

5.3 Feedback Loop

[Calibration dataset] → [Run extractor] → [Compare to hand labels]
  → [Confusion matrix] → [Adjust type descriptions]
  → [Re-run] → [Measure improvement]
  → [When stable: promote descriptions to config.yaml]

This is not a one-time exercise. As new source types are added or existing schemas are used on new document types, the calibration dataset should grow.

5.4 Governance

Type changes require: Updated description in config.yaml + re-run against calibration dataset + update to concept-taxonomy-reference.md
New type additions require: Justification against MECE principle (why existing types don't cover it) + calibration dataset entries
Type removals require: Evidence from extraction logs that the type is used <2% of the time or is never the primary classification
Coherence relationship changes require: Re-classification of existing coherence data against the new dimensional schema
Review cadence: After every 5 new ingestion jobs, review type distribution and confusion patterns

6. Implementation Phases

Phase 0 — Baseline Measurement (do this first)

Goal: Measure extraction quality before making any changes, so we know what improved and what didn't.

Tasks:

Create validation/calibration/ directory structure
Select 3-5 documents from completed jobs (prefer already-ingested documents across multiple schemas)
Hand-annotate 50-100 concepts per document — primary annotation by Atlas, 20% subset verified by a second annotator
Build comparison script: scripts/validate_extraction.py that takes extractor output and calibration data, produces confusion matrix and per-type metrics
Run the current extractor against the calibration dataset — no changes to config, no new prompts, just measure what we have
Record baseline metrics in validation/baseline_metrics.md

Deliverable: Calibration dataset, validation script, baseline metrics. These are the yardstick every subsequent change is measured against.

Estimated effort: Medium — annotation labor is the bottleneck (150-500 concepts). The script is straightforward.

Why this comes first: Without a baseline, Phase 1 changes are invisible — we won't know if they helped or hurt. Running baseline first means every subsequent phase has a number to beat.

Phase 1 — Enhanced Type Specs & Prompt Engineering

Goal: Upgrade the type specification format, add decision trees, rename one type, assemble prompts properly. The biggest accuracy gains per hour of work.

Tasks:

Add scope field to each of the 7 schemas in config.yaml
Add long_description, linguistic_cues, disambiguation, prototypical_examples, priority to each concept type in each schema
Rename contradiction → counterargument in book schema
Change instruction_manual default type from tip to step
Add source_position to concept metadata (optional — populated when section structure is available)
Update extractor prompt template to include binary decision tree for schemas with adjacent-type ambiguity (book, academic_paper, article)
Update prompt assembly logic to include long_description, linguistic_cues, disambiguation, and prototypical_examples for each type (see §4.4)
Bulk-update any existing contradiction-typed concepts in Chroma to counterargument (conditional — only if such concepts exist)
Update docs/specs/concept-taxonomy-reference.md to reflect all changes

Deliverable: Updated config.yaml with enhanced type specs, updated prompt template, updated reference doc, migrated legacy concepts.

Estimated effort: Small — mostly YAML editing and prompt engineering.

Phase 2 — Section-Aware Extraction

Goal: Pass the structural context the pipeline already produces into the extractor prompt, so the LLM can use position, chapter titles, and section headings to inform type selection.

Tasks:

Identify where in the pipeline the extraction prompt is assembled
Wire section metadata (chapter title, section heading, position fraction) from structure.json and summaries into the prompt builder
Add the section context block to the extraction prompt template (see §4.6 for format)
Test on 1-2 existing documents to verify the context appears correctly
Update docs/specs/concept-taxonomy-reference.md to document the new prompt structure

Deliverable: Section-aware extraction prompts using existing pipeline metadata.

Estimated effort: Small-medium — the data exists, this is wiring and prompt adjustment.

Phase 3 — Re-Measure & Iterate

Goal: Run the calibrated system against the same baseline dataset. Compare to Phase 0 numbers. Tune until ≥85% agreement.

Tasks:

Run extractor with Phase 1+2 changes against the calibration dataset
Compare results to Phase 0 baseline — which types improved, which got worse, which are unchanged
Generate confusion matrix
For each confused pair, either: (a) sharpen descriptions with more specific disambiguation rules, or (b) if confusion persists across iterations, consider merging the types
Re-run and re-measure
When stable, finalize descriptions in config.yaml
Record final metrics in validation/final_metrics.md
Re-index any existing ingested concepts that were extracted with the old type specs (optional — may not be worth it if changes are description-only and the total data is small)

Deliverable: Finalized type specs with measured quality metrics. Before/after comparison.

Estimated effort: Medium — depends on how many iterations are needed.

Phase 4 — Coherence Relationship Redesign

Goal: Replace the flat 7-type coherence vocabulary with the dimensional classification system (§4.7), enabling richer relationship expression and better graph reasoning.

Tasks:

Define the coherence dimensional schema in config.yaml (alignment, function, scope)
Define the mapping from dimensional combinations to primary labels (for backward compatibility)
Update the coherence extraction prompt to classify along each dimension independently
Update the coherence storage schema in graph.db to store dimensional scores alongside the derived label
Re-classify existing coherence relationships by running the new prompt against stored concept pairs (batch operation)
Update any downstream code that reads coherence relationships to handle the new format
Update docs/specs/concept-taxonomy-reference.md to document the new coherence types

Deliverable: Dimensional coherence classification with backward-compatible labels, migrated existing data.

Estimated effort: Medium — the prompt and schema changes are straightforward; the re-classification batch job is the main work.

Why this comes after Phase 3: The coherence layer depends on concept types being stable. If we change the concept taxonomy and the coherence vocabulary simultaneously, we can't tell which change caused any observed improvement or regression.

Phase 5 — Extend (optional, post-validation)

Goal: Apply validated patterns to new source types, add diagnostic infrastructure if needed, explore advanced features.

Tasks:

If adding new source types, use the enhanced type specification format from Phase 1
Add calibration entries for new source types before deploying
If calibration revealed systematic confusion that needs deeper diagnosis, add extraction_rationale and adjacent_type_score as optional metadata fields — but only if there's a specific problem to solve
Consider adding source_position-based extraction rules (e.g., in academic_paper, weight hypothesis higher in introduction sections, finding higher in results sections) — now that we have calibration data to measure whether position-awareness improves accuracy
Build a coherence graph visualization or analysis tool (optional — would help with qualitative review of relationship quality)

Deliverable: Extension-ready architecture with ongoing measurement.

7. Open Questions

These are decisions that should be made before or during Phase 1, but don't block starting Phase 0:

Should we add source_position as a required extraction field? Currently proposed as optional in Phase 1. Making it required means every extraction includes section-aware position, which helps with hypothesis/finding disambiguation in academic papers. But it depends on the parsed document providing section labels consistently. Recommendation: start optional, require it if Phase 3 calibration shows position would have resolved confusions.
What's the threshold for creating a new schema vs. extending an existing one? If we ingest a new document type (e.g., podcast_transcript), should it get its own schema or use the closest existing one? Recommendation: start by mapping to the closest existing schema, and only create a new schema if calibration shows ≥30% of concepts don't fit any existing type cleanly.
Should the coherence dimensional scores be stored as separate fields or as a structured object? Separate fields are simpler to query in SQLite. A structured JSON object is more flexible. Recommendation: separate fields for the three dimensions (alignment, function, scope) plus the derived label, since these are small and enumerable.
What's the priority order between Phase 2 (section-aware) and Phase 4 (coherence)? Phase 2 is smaller and directly supports Phase 3 measurement. Phase 4 is independent and can run in parallel if resources allow. Recommendation: sequential (Phase 2 → Phase 3 → Phase 4) unless you have two developers.

8. What This Plan Does NOT Change

To be explicit about stability:

Pipeline architecture — parse → structure → summarize → classify → extract → index → coherence is preserved. We are changing what goes into the prompts, not how the pipeline flows.
7 source types — keeping all 7 (book, academic_paper, article, technical_document, instruction_manual, legal_document, fiction)
7 schemas — keeping the one-schema-per-source-type architecture (confirmed by Semantic Arts' modular design principle)
Type counts — staying at 6-7 types per schema (one rename, no additions or removals in Phase 1)
Chroma collections — ingested_concepts, knowledge_base, agent_memory structure preserved (Gitea #38 may address this separately)
Classifier → Extractor split — the two-pass LLM architecture (classify schema, then extract concepts) is correct and stays

What does change:

Prompt content and structure — richer type specs, decision trees, section context
Config format — from flat type labels to structured type specifications
Coherence vocabulary — from flat 7-type list to dimensional classification
Measurement — from "we don't know" to a baseline number and a target

9. References

Rosch, E. (1978). Principles of Categorization. — Basic-level categories, prototypicality, cognitive economy
Gozli, D. (2023). On the Relative Nature of Categories. — Sense vs. choice categories, deviations from basic-level
Mammen, J. — Sense vs. choice categories (as discussed in Gozli 2023)
NNGroup / Taxon.dev — MECE taxonomy design, 10-step process, consistent labels
Semantic Arts — 8 ontology design schools, modular design, scope definition
Visser, S. et al. (2021). Argument Scheme Annotation. — Inter-annotator agreement, Walton's 60 schemes, Wagemans' Periodic Table, decision tree heuristics, content vs. function distinction
Walton, D. — 60 argument schemes (empirically-motivated classification)
Wagemans, J. — Periodic Table of Arguments (36 types, factorial typology)
LandingAI (2025). LLM Extraction Schema Design. — Descriptive field names, document-mirroring, data type specificity

Appendix A: Quick-Reference — Changes Summary

What	Current	Proposed	Phase
Measurement	None	Baseline calibration dataset + comparison script + metrics	0
`book.contradiction`	`contradiction`	Rename to `counterargument`	1
`instruction_manual` default	`tip`	Change to `step`	1
Type descriptions	1-sentence label	Structured spec (long desc + cues + disambiguation + examples + priority)	1
Schema scope	Implicit (schema name)	Explicit `scope` field	1
Extractor prompt	Type list only	Type list + binary decision tree for ambiguous schemas	1
Prompt assembly	Not specified	Explicit field ordering, token budget, example format (§4.4)	1
Concept metadata	Current fields	Add `source_position` (optional)	1
Legacy data migration	Not addressed	Bulk-update `contradiction` → `counterargument` in Chroma	1
Section context	Not passed to extractor	Chapter title, section heading, position fraction in prompt	2
Re-measurement	N/A	Run against calibration dataset, compare to baseline, iterate to ≥85%	3
Coherence relationships	Flat 7-type list	Dimensional classification (alignment × function × scope) with derived labels	4
Coherence migration	N/A	Re-classify existing relationships via batch prompt	4
Quality target	Not measured	≥85% type agreement vs. hand labels	3
Diagnostic metadata	Not applicable	`extraction_rationale`, `adjacent_type_score` — only if calibration reveals need	5

Appendix B: Example Enhanced Type Spec — `book.claim`

claim:
  label: "Claim"
  short_description: "The author's own assertion, thesis, or position"
  long_description: |
    A declarative statement that the author argues for, defends, or puts forward as true.
    This is the author's voice making a substantive case. Claims are the primary knowledge
    atoms in non-fiction — they are what a reader would quote or reference when discussing
    the book's arguments.
  linguistic_cues:
    - "I argue that..."
    - "The evidence shows..."
    - "It is clear that..."
    - "We will see that..."
    - "The key insight is..."
    - Declarative sentences in the author's voice that assert something as true
  disambiguation:
    - "NOT a definition of a term (use 'definition' — definitions explain what something means, claims assert that something is true)"
    - "NOT a cited external fact (use 'evidence' — claims are the author's own, evidence is borrowed)"
    - "NOT a challenge to a specific alternative (use 'counterargument' — counterarguments explicitly target another view)"
    - "NOT a prescriptive rule or guideline (use 'principle' — principles tell you what to do or what generally applies, claims describe what is true)"
  prototypical_examples:
    - "System 1 operates automatically and quickly with little or no effort and no sense of voluntary control."
    - "The planning fallacy describes a tendency for people to underestimate the time, costs, and risks of future actions."
    - "Expert intuition is more reliable than we think, but only in environments with regular feedback."
  priority: 1

Appendix C: Example Coherence Dimensional Classification

Old format:

{
  "source_concept": "concept_abc123",
  "target_concept": "concept_def456",
  "relationship": "agrees"
}

New format:

{
  "source_concept": "concept_abc123",
  "target_concept": "concept_def456",
  "alignment": "supportive",
  "function": "warrants",
  "scope": "same",
  "derived_label": "supports",
  "confidence": 0.82
}

Derived label rules (complete):

alignment	function	scope	derived_label
supportive	corroborates	same	`same_idea`
supportive	corroborates	broader/narrower	`agrees`
supportive	warrants	any	`supports`
supportive	exemplifies	narrower	`exemplifies`
supportive	specializes	narrower	`specializes`
supportive	extends	broader	`extends`
opposing	rebuts	any	`contradicts`
opposing	undermines	any	`undermines`
opposing	qualifies	any	`qualifies`
neutral	extends	broader	`extends`
neutral	specializes	narrower	`specializes`

Appendix D: Phase Dependency Graph

Phase 0 (Baseline)
    ↓
Phase 1 (Type Specs + Prompts)
    ↓
Phase 2 (Section-Aware Extraction)
    ↓
Phase 3 (Re-Measure + Iterate) ← feeds back to Phase 1 if needed
    ↓
Phase 4 (Coherence Redesign) ← independent of Phase 2, could parallel
    ↓
Phase 5 (Extend) ← only if needed

Critical path: Phase 0 → Phase 1 → Phase 2 → Phase 3. These must be sequential because each depends on the previous.

Parallelizable: Phase 4 (coherence) is independent of Phases 2-3 and could run in parallel with a separate developer if resources allow. The only constraint is that concept types should be stable (Phase 3 complete or near-complete) before finalizing coherence, since coherence relationships reference concept types.

57 KiB Raw Permalink Blame History Unescape Escape

Saga Concept Taxonomy & Extraction Schema — Redesign Plan

1. Purpose & Scope

2. Research Summary — What the Science Says

2.1 Cognitive Science Foundations (Rosch 1978, Gozli 2023)

2.2 Taxonomy & Ontology Design Best Practices

2.3 Ontology Design Schools (Semantic Arts)

2.4 Argumentation Theory (Visser et al. 2021, Walton, Wagemans)

2.5 Extraction Schema Design for LLMs (LandingAI 2025)

3. Current State Analysis — Where the Seams Are

3.1 MECE Audit of Each Schema

book schema — issues found

academic_paper schema — issues found

article schema — issues found

technical_document schema — no issues

instruction_manual schema — no issues

legal_document schema — no issues

fiction schema — minor issue

3.2 Type Ordering Audit

3.3 Description Quality Audit

3.4 Coherence Relationship Audit

4. Architecture Design — What to Build

4.1 Enhanced Concept Type Specification Format

4.2 Schema Scope Statements

4.3 Decision Tree for Disambiguation

4.4 Prompt Assembly

4.5 Concept Metadata Schema Additions

4.6 Section-Aware Extraction

4.7 Coherence Relationship Redesign

5. Validation & Quality Measurement Plan

5.1 Calibration Dataset

5.2 Inter-Rater Reliability Metric

5.3 Feedback Loop

5.4 Governance

6. Implementation Phases

Phase 0 — Baseline Measurement (do this first)

Phase 1 — Enhanced Type Specs & Prompt Engineering

Phase 2 — Section-Aware Extraction

Phase 3 — Re-Measure & Iterate

Phase 4 — Coherence Relationship Redesign

Phase 5 — Extend (optional, post-validation)

7. Open Questions

8. What This Plan Does NOT Change

9. References

Appendix A: Quick-Reference — Changes Summary

Appendix B: Example Enhanced Type Spec — book.claim

Appendix C: Example Coherence Dimensional Classification

Appendix D: Phase Dependency Graph

57 KiB

Raw Permalink Blame History

`book` schema — issues found

`academic_paper` schema — issues found

`article` schema — issues found

`technical_document` schema — no issues

`instruction_manual` schema — no issues

`legal_document` schema — no issues

`fiction` schema — minor issue

Appendix B: Example Enhanced Type Spec — `book.claim`