Update: docs/specs/saga-taxonomy-redesign-plan.md — Expanded scope from tuning pass to full redesign: added Phase 0 baseline measurement, section-aware extraction (§4.6), coherence layer redesign (§4.7), reordered phases, updated appendices, and revised §1/§8 to reflect the broader mandate.
This commit is contained in:
Atlas
@@ -4,27 +4,25 @@ id: b47806bc-e187-4381-bbd2-b9d5e71ea37a
|
||||
keywords:
|
||||
- schema
|
||||
- concept
|
||||
- extraction
|
||||
- type
|
||||
- saga
|
||||
- extraction
|
||||
- types
|
||||
- phase
|
||||
- claim
|
||||
- prompt
|
||||
- coherence
|
||||
- taxonomy
|
||||
- author
|
||||
- principle
|
||||
- plan
|
||||
- calibration
|
||||
- scope
|
||||
- extractor
|
||||
- prompt
|
||||
- as
|
||||
links: []
|
||||
modified: '2026-04-14T21:21:32+00:00'
|
||||
summary: Architecture and implementation plan for refining Saga's concept taxonomy
|
||||
and extraction schemas based on scientific research into cognitive categorization,
|
||||
ontology design, and LLM extraction best practices. Covers MECE compliance, type
|
||||
descriptions, schema structure, and validation approach. Revised to fix disambiguation
|
||||
logic, add prompt assembly spec, restructure decision tree, and defer speculative
|
||||
metadata.
|
||||
modified: '2026-04-14T21:35:04+00:00'
|
||||
summary: Comprehensive redesign plan for Saga's concept taxonomy, extraction schemas,
|
||||
and coherence relationships. Covers baseline measurement, enhanced type specs, section-aware
|
||||
extraction, coherence layer redesign based on argumentation theory, and validation
|
||||
infrastructure. Expanded from tuning pass to full redesign scope.
|
||||
tags:
|
||||
- spec
|
||||
- saga
|
||||
@@ -39,7 +37,7 @@ title: Saga Concept Taxonomy & Extraction Schema — Redesign Plan
|
||||
# Saga Concept Taxonomy & Extraction Schema — Redesign Plan
|
||||
|
||||
**Author:** Atlas (Librarian & Historian)
|
||||
**Date:** 2025-07-12 (revised 2025-07-13)
|
||||
**Date:** 2025-07-12 (revised 2025-07-13, expanded 2025-07-14)
|
||||
**Status:** Proposed — awaiting developer review
|
||||
**Tracks:** Gitea #37 (taxonomy refinement), Gitea #38 (collection architecture redesign)
|
||||
**Research basis:** Hermione `rs_concept_taxonomy_science` — 51 extracted concepts across 4 chapters
|
||||
@@ -48,20 +46,26 @@ title: Saga Concept Taxonomy & Extraction Schema — Redesign Plan
|
||||
|
||||
## 1. Purpose & Scope
|
||||
|
||||
This plan provides a grounded, phased approach for tuning Saga's concept taxonomy and extraction schemas from their current working state to a state backed by established research in cognitive science, ontology engineering, and LLM extraction design.
|
||||
This plan provides a grounded, phased approach for improving Saga's concept extraction quality — from the type definitions that guide extraction, through the prompts the LLM sees, to the coherence relationships that connect extracted concepts across documents.
|
||||
|
||||
The original plan was scoped as a tuning pass. After analysis, the scope has been expanded to include three additional high-ROI areas that the research supports and that are cheap to do now (4 completed jobs, small data footprint) but expensive to retrofit later.
|
||||
|
||||
**What this covers:**
|
||||
- Concept type definitions (the "what we extract" layer)
|
||||
- Concept type descriptions (the "how to recognize it" layer)
|
||||
- Schema structure and type ordering (the "how we organize it" layer)
|
||||
- Validation and quality measurement (the "how we know it works" layer)
|
||||
- Baseline measurement — knowing where we stand before changing anything
|
||||
- Concept type definitions — the "what we extract" layer
|
||||
- Concept type descriptions — the "how to recognize it" layer
|
||||
- Prompt assembly — how type specs become LLM instructions
|
||||
- Section-aware extraction — using document structure the pipeline already produces
|
||||
- Schema structure and type ordering — the "how we organize it" layer
|
||||
- Coherence relationship redesign — the "how concepts relate" layer
|
||||
- Validation and quality measurement — the "how we know it works" layer
|
||||
|
||||
**What this does NOT cover:**
|
||||
- Pipeline architecture changes (parsing, chunking, summarization — separate work)
|
||||
- Coherence graph redesign (related but distinct)
|
||||
- Chroma collection restructuring (Gitea #38 — this plan feeds into it)
|
||||
- New source type or schema additions (future work, governed by §5.4)
|
||||
|
||||
**Design philosophy:** We are not rebuilding — we are tuning. The current 7-source-type, 7-schema architecture is structurally sound. The research confirms the direction and identifies specific seams that need tightening.
|
||||
**Design philosophy:** The pipeline architecture — parse → structure → summarize → classify → extract → index → coherence — is structurally sound. The 7-source-type, 7-schema modular design is exactly what the research recommends. We are not rebuilding the pipeline. We are rebuilding the *prompts, the metadata, and the relationship vocabulary* that the pipeline uses.
|
||||
|
||||
---
|
||||
|
||||
@@ -142,7 +146,7 @@ Eight schools of ontology design were identified. The two most relevant to Saga:
|
||||
|
||||
**Implication for Saga:** Each schema should carry an explicit scope statement describing what it's optimized to extract. Currently the schema name (`book`, `academic_paper`) implies the scope, but an explicit purpose statement would improve extractor performance.
|
||||
|
||||
### 2.4 Argumentation Theory (Visser et al. 2021)
|
||||
### 2.4 Argumentation Theory (Visser et al. 2021, Walton, Wagemans)
|
||||
|
||||
**Principle — Classification difficulty is real:** Even trained annotators achieved only moderate inter-annotator agreement when classifying arguments into established typologies (Walton's 60 schemes, Wagemans' 36 types). Many arguments resisted classification entirely.
|
||||
|
||||
@@ -161,6 +165,19 @@ Eight schools of ontology design were identified. The two most relevant to Saga:
|
||||
|
||||
**Implication for Saga:** The extractor prompt should include a binary decision tree where each fork asks a single yes/no question. The tree should front-load the most common types so the extractor reaches the right answer quickly. See §4.3 for the full specification.
|
||||
|
||||
**Principle — Content vs. function distinction:** In argumentation theory, what a proposition IS (its content) is distinct from what it DOES (its function in an argument). A statistic can be a *claim* (when the author asserts it) or *evidence* (when the author uses it to support something else). The same content plays different functional roles.
|
||||
|
||||
> *"The content vs function distinction is critical: what a proposition IS vs what it DOES in the argument."* — Visser et al. (2021)
|
||||
|
||||
**Implication for Saga:** This distinction is exactly what our coherence relationship types should capture. Currently `agrees` and `contradicts` describe *stance* (positive vs. negative), but they don't capture *function* (supporting, warranting, rebutting, qualifying). The coherence layer should be redesigned to reflect argument function, not just sentiment alignment. See §4.7.
|
||||
|
||||
**Principle — Walton's 60 schemes and Wagemans' 36 types:** These are comprehensive typologies of argument *schemes* — the patterns of reasoning that connect premises to conclusions. They include categories like "argument from example," "argument from cause to effect," "practical reasoning," and "ad hominem variants."
|
||||
|
||||
> *"Walton's taxonomy: empirically-motivated classification with 60 upper-level argument schemes"* — Visser et al. (2021)
|
||||
> *"Wagemans' Periodic Table of Arguments: a priori exhaustive factorial typology comprising 36 argument types"* — Visser et al. (2021)
|
||||
|
||||
**Implication for Saga:** We should not adopt 60 or 36 relationship types — that would be absurd for our use case. But the *dimensional structure* of these typologies is instructive. Wagemans' factorial approach (statement type × assertion type × subject matter) suggests that our coherence relationships could be organized along a small number of orthogonal dimensions rather than as a flat list of 7 labels. See §4.7.
|
||||
|
||||
### 2.5 Extraction Schema Design for LLMs (LandingAI 2025)
|
||||
|
||||
**Principle — Descriptive field names with contextual descriptions:** Field names must clearly indicate what to extract. Descriptions should specify format, currency, length constraints, and examples.
|
||||
@@ -176,7 +193,9 @@ Eight schools of ontology design were identified. The two most relevant to Saga:
|
||||
|
||||
> *"organize schema to reflect how data appears in parsed Markdown and how it will be used; use nested objects for hierarchical data; match document flow order"* — LandingAI (2025)
|
||||
|
||||
**Implication for Saga:** Within each schema, types should be ordered by likelihood of occurrence in that source type. For example, in `academic_paper`, `finding` should come first (it's the most common and important), not `hypothesis`. In `book`, `claim` first is correct. In `fiction`, `theme` first is correct.
|
||||
**Implication for Saga:** Two implications:
|
||||
1. Within each schema, types should be ordered by likelihood of occurrence in that source type. For example, in `academic_paper`, `finding` should come first (it's the most common and important), not `hypothesis`. In `book`, `claim` first is correct.
|
||||
2. The extraction prompt should receive section-level context — the chapter title, section heading, and position in the document — not just the raw text. The pipeline already produces this information during the structure and summarize phases. We should pass it through to extraction. See §4.6.
|
||||
|
||||
**Principle — Strict data type specification:** Prevent downstream conversion errors by specifying correct types.
|
||||
|
||||
@@ -203,7 +222,7 @@ Eight schools of ontology design were identified. The two most relevant to Saga:
|
||||
|
||||
| Problem | Types | Why it's not MECE | Recommended resolution |
|
||||
|---|---|---|---|
|
||||
| `hypothesis` and `finding` can overlap | hypothesis, finding | A confirmed hypothesis is reported as a finding. The same statement can be both depending on where in the paper it appears (intro = hypothesis, results = finding). | **Disambiguate by tense and framing, not by position.** `hypothesis` = "a prediction or proposition framed in future or conditional tense about what the study expects to find." `finding` = "an observed result framed in past tense about what the study found." Key test: is the author predicting what will happen (hypothesis) or reporting what did happen (finding)? Position in the document can be a *corroborating signal* if available, but tense/framing is the primary disambiguation criterion and works without structural metadata. |
|
||||
| `hypothesis` and `finding` can overlap | hypothesis, finding | A confirmed hypothesis is reported as a finding. The same statement can be both depending on where in the paper it appears (intro = hypothesis, results = finding). | **Disambiguate by tense and framing, not by position.** `hypothesis` = "a prediction or proposition framed in future or conditional tense about what the study expects to find." `finding` = "an observed result framed in past tense about what the study found." Key test: is the author predicting what will happen (hypothesis) or reporting what did happen (finding)? Position in the document can be a *corroborating signal* if available (§4.6), but tense/framing is the primary disambiguation criterion and works without structural metadata. |
|
||||
| `contribution` is orphaned | contribution | Contributions are often a mix of findings, methods, and frameworks. They rarely appear as standalone knowledge atoms. | **Keep but narrow.** `contribution` = "a specific claimed novelty or advance that the authors explicitly identify as their contribution, typically in the introduction or discussion." |
|
||||
|
||||
#### `article` schema — issues found
|
||||
@@ -250,6 +269,22 @@ Current default types (the type the extractor falls back to) are generally corre
|
||||
|
||||
Current descriptions are single-sentence labels. Per LandingAI's research, they should be expanded to include linguistic cues, disambiguation rules, and prototypical examples. All 7 schemas need this treatment. See §4.1 for the specification.
|
||||
|
||||
### 3.4 Coherence Relationship Audit
|
||||
|
||||
The current 7 coherence relationship types were designed well intuitively, but they were not grounded in the argumentation theory research. Reviewing them against the research:
|
||||
|
||||
| Relationship | Current description | Assessment |
|
||||
|---|---|---|
|
||||
| `same_idea` | Two concepts express the same underlying idea | ✓ Clear, no ambiguity |
|
||||
| `agrees` | One concept supports or corroborates another | ⚠️ Underspecified — "supports" and "corroborates" are different functions. Corroboration means independent evidence arriving at the same conclusion. Support means one concept provides reasoning for another. These are distinct argumentative functions. |
|
||||
| `contradicts` | One concept opposes or refutes another | ⚠️ Underspecified — "opposes" and "refutes" are different. One can oppose something without refuting it (disagreement vs. disproof). |
|
||||
| `extends` | One concept builds upon or elaborates another | ✓ Clear |
|
||||
| `specializes` | One concept is a specific case of a more general one | ✓ Clear |
|
||||
| `exemplifies` | One concept provides a concrete example of another | ✓ Clear |
|
||||
| `undermines` | One concept weakens the basis for another without directly contradicting it | ✓ Clear — this is actually the most nuanced current type |
|
||||
|
||||
**Key finding:** The positive alignment types (`agrees`) and negative alignment types (`contradicts`) conflate *stance* with *function*. Argumentation theory distinguishes between these dimensions. See §4.7 for the proposed redesign.
|
||||
|
||||
---
|
||||
|
||||
## 4. Architecture Design — What to Build
|
||||
@@ -375,6 +410,7 @@ If a decision tree is used (§4.3), it appears *after* the type definitions and
|
||||
|
||||
```
|
||||
[Schema scope statement]
|
||||
[Section context — chapter title, section heading, position] (see §4.6)
|
||||
[Type definitions — ordered by priority]
|
||||
[Decision tree for disambiguation]
|
||||
[Extraction instructions — format, output schema, constraints]
|
||||
@@ -418,16 +454,127 @@ concept_metadata:
|
||||
|
||||
`source_position` enables position-aware disambiguation as a corroborating signal (e.g., in `academic_paper`, statements in a "results" section are more likely to be `finding`). It is optional in Phase 1 — populated when the section structure is available from the parsed document.
|
||||
|
||||
**Phase 4 — optional diagnostic fields:**
|
||||
**Phase 5 — optional diagnostic fields:**
|
||||
|
||||
These are deferred to Phase 4 and included only if calibration reveals systematic confusion that needs debugging:
|
||||
These are deferred to Phase 5 and included only if calibration reveals systematic confusion that needs debugging:
|
||||
|
||||
```yaml
|
||||
extraction_rationale: # Short explanation of why this type was chosen (string, optional)
|
||||
adjacent_type_score: # If the extractor considered a second type, record it (string, optional)
|
||||
```
|
||||
|
||||
**Why these are deferred:** `extraction_rationale` adds prompt overhead for every extracted concept, and LLMs are known to confabulate post-hoc justifications rather than accurately reporting their reasoning. `adjacent_type_score` asks the LLM to quantify its own uncertainty — something LLMs are notably poor at. Neither field is used by the validation plan (§5), which compares against hand labels. Their value is purely for post-hoc debugging of systematic confusion patterns. They should only be added if Phase 2-3 calibration reveals a specific problem they would help diagnose.
|
||||
**Why these are deferred:** `extraction_rationale` adds prompt overhead for every extracted concept, and LLMs are known to confabulate post-hoc justifications rather than accurately reporting their reasoning. `adjacent_type_score` asks the LLM to quantify its own uncertainty — something LLMs are notably poor at. Neither field is used by the validation plan (§5), which compares against hand labels. Their value is purely for post-hoc debugging of systematic confusion patterns. They should only be added if Phase 3-4 calibration reveals a specific problem they would help diagnose.
|
||||
|
||||
### 4.6 Section-Aware Extraction
|
||||
|
||||
The pipeline already produces rich structural context during the `structure` and `summarize` phases. Currently, the extractor receives the raw section text and the schema type list, but *not* the chapter title, section heading, or position-in-document metadata. The LandingAI research and common sense both say this matters: a paragraph in "Chapter 3: The Anatomy of Bias" should be extracted differently than one in "Chapter 18: About the Author."
|
||||
|
||||
**What to pass into the extraction prompt:**
|
||||
|
||||
```yaml
|
||||
section_context:
|
||||
document_title: "Thinking, Fast and Slow"
|
||||
chapter_title: "The Characters of the System"
|
||||
chapter_number: 1
|
||||
section_title: "An Autonomous System"
|
||||
section_position: "early" # early/middle/late within the document
|
||||
document_position: 0.05 # fraction of total document (0.0 to 1.0)
|
||||
```
|
||||
|
||||
**How this is assembled into the prompt:**
|
||||
|
||||
The section context block appears at the top of the extraction prompt, before the type definitions:
|
||||
|
||||
```
|
||||
You are extracting concepts from:
|
||||
Document: "Thinking, Fast and Slow"
|
||||
Chapter 1: "The Characters of the System"
|
||||
Section: "An Autonomous System"
|
||||
Position: Early in the document (5% through)
|
||||
|
||||
[Schema scope statement]
|
||||
[Type definitions — ordered by priority]
|
||||
[Decision tree for disambiguation]
|
||||
[Extraction instructions]
|
||||
```
|
||||
|
||||
**How this improves extraction:**
|
||||
|
||||
1. **Position priors:** Early in a book, concepts are more likely to be `definition` (introducing terms) or `framework` (establishing the author's model). Late in a book, concepts are more likely to be `claim` (arguing specific positions) or `example` (illustrating established ideas). The extractor doesn't need explicit rules for this — just seeing the position is enough for the LLM to adjust its priors.
|
||||
|
||||
2. **Section title cues:** A section titled "Experimental Evidence" primes the extractor to look for `finding`-type concepts. A section titled "What This Means" primes for `claim` or `principle`. Again, no explicit rules needed — the LLM picks up on these cues naturally.
|
||||
|
||||
3. **Academic paper disambiguation:** For `academic_paper` schemas, the section context directly resolves the hypothesis/finding ambiguity. Statements in "Introduction" or "Hypotheses" sections are overwhelmingly `hypothesis`; statements in "Results" sections are overwhelmingly `finding`.
|
||||
|
||||
**Implementation:**
|
||||
|
||||
The structural information is already available in `structure.json` and the summary files. The change is:
|
||||
- The extractor prompt builder reads section metadata from the existing pipeline outputs
|
||||
- Injects it as a context block at the top of the prompt
|
||||
- No changes to parsing or chunking
|
||||
|
||||
### 4.7 Coherence Relationship Redesign
|
||||
|
||||
The current 7 coherence types (`same_idea`, `agrees`, `contradicts`, `extends`, `specializes`, `exemplifies`, `undermines`) capture useful relationships but conflate two independent dimensions: **stance** (positive vs. negative alignment) and **function** (what role the relationship plays in the argumentative structure).
|
||||
|
||||
The argumentation theory research (Visser et al., Walton, Wagemans) provides a clearer vocabulary. The key insight from Wagemans is the *factorial* approach: relationship types can be described as combinations of a small number of independent dimensions rather than as a flat list.
|
||||
|
||||
**Proposed dimensional structure:**
|
||||
|
||||
The coherence relationship between two concepts should be described along three dimensions:
|
||||
|
||||
| Dimension | Values | What it captures |
|
||||
|---|---|---|
|
||||
| **Alignment** | `supportive`, `opposing`, `neutral` | Does concept B strengthen, weaken, or not affect concept A? |
|
||||
| **Function** | `corroborates`, `warrants`, `exemplifies`, `specializes`, `extends`, `rebuts`, `undermines`, `qualifies` | What is the argumentative role of the relationship? |
|
||||
| **Scope** | `same`, `broader`, `narrower` | Is B at the same, higher, or lower level of abstraction than A? |
|
||||
|
||||
**How this maps to the current 7 types:**
|
||||
|
||||
| Current type | New representation |
|
||||
|---|---|
|
||||
| `same_idea` | alignment=`supportive`, function=`corroborates`, scope=`same` |
|
||||
| `agrees` | alignment=`supportive`, function=`corroborates` or `warrants`, scope=any |
|
||||
| `contradicts` | alignment=`opposing`, function=`rebuts`, scope=`same` |
|
||||
| `extends` | alignment=`supportive`, function=`extends`, scope=`broader` |
|
||||
| `specializes` | alignment=`supportive`, function=`specializes`, scope=`narrower` |
|
||||
| `exemplifies` | alignment=`supportive`, function=`exemplifies`, scope=`narrower` |
|
||||
| `undermines` | alignment=`opposing`, function=`undermines`, scope=any |
|
||||
|
||||
**What this adds (the new relationships the current system can't express):**
|
||||
|
||||
| New function | Description | Example |
|
||||
|---|---|---|
|
||||
| `warrants` | B provides the reasoning or evidence that justifies A | "Regular feedback improves performance" *warrants* "Expert intuition is reliable in structured environments" |
|
||||
| `rebuts` | B directly contradicts A's conclusion | "System 1 is always unreliable" *rebuts* "Expert intuition is reliable in structured environments" |
|
||||
| `qualifies` | B adds a condition, limitation, or scope restriction to A | "Only in environments with regular feedback" *qualifies* "Expert intuition is reliable" |
|
||||
|
||||
**Implementation approach:**
|
||||
|
||||
The coherence phase of the pipeline currently uses an LLM to classify relationships into the 7 flat types. The change is:
|
||||
|
||||
1. **Replace the flat classification with dimensional classification.** The prompt asks the LLM to classify along each dimension independently. This is easier for the LLM (each dimension is a simple choice) and produces richer output.
|
||||
|
||||
2. **Derive a primary relationship label from the dimensional scores.** For backward compatibility and human readability, derive a label from the combination:
|
||||
```
|
||||
if alignment == supportive and function == corroborates and scope == same → "same_idea"
|
||||
if alignment == supportive and function == warrants → "supports"
|
||||
if alignment == opposing and function == rebuts → "contradicts"
|
||||
if alignment == opposing and function == undermines → "undermines"
|
||||
if alignment == opposing and function == qualifies → "qualifies"
|
||||
... etc.
|
||||
```
|
||||
|
||||
3. **Store both the dimensional scores and the derived label.** The label is used for display and filtering. The dimensional scores are used for graph analysis and agent reasoning.
|
||||
|
||||
4. **Migration of existing coherence data:** Existing relationships in `coherence/graph.db` can be re-classified by running the new dimensional prompt against the stored concept pairs. This is a batch operation, not a manual migration.
|
||||
|
||||
**Why this is worth doing now:**
|
||||
|
||||
- The coherence graph is the feature that makes the KB more than a bag of concepts — it's what enables agents to reason about how ideas connect across documents.
|
||||
- The current types were designed intuitively and not revisited. The research provides a principled vocabulary.
|
||||
- There are only 4 completed jobs' worth of coherence data to migrate — the cost is minimal.
|
||||
- Delaying this means more data to migrate later and more downstream code that depends on the current flat type list.
|
||||
|
||||
---
|
||||
|
||||
@@ -485,15 +632,34 @@ This is not a one-time exercise. As new source types are added or existing schem
|
||||
- **Type changes require:** Updated description in `config.yaml` + re-run against calibration dataset + update to `concept-taxonomy-reference.md`
|
||||
- **New type additions require:** Justification against MECE principle (why existing types don't cover it) + calibration dataset entries
|
||||
- **Type removals require:** Evidence from extraction logs that the type is used <2% of the time or is never the primary classification
|
||||
- **Coherence relationship changes require:** Re-classification of existing coherence data against the new dimensional schema
|
||||
- **Review cadence:** After every 5 new ingestion jobs, review type distribution and confusion patterns
|
||||
|
||||
---
|
||||
|
||||
## 6. Implementation Phases
|
||||
|
||||
### Phase 1 — Foundation (config.yaml changes)
|
||||
### Phase 0 — Baseline Measurement (do this first)
|
||||
|
||||
**Goal:** Upgrade the type specification format and rename one type, without changing any structural boundaries.
|
||||
**Goal:** Measure extraction quality *before* making any changes, so we know what improved and what didn't.
|
||||
|
||||
**Tasks:**
|
||||
1. [ ] Create `validation/calibration/` directory structure
|
||||
2. [ ] Select 3-5 documents from completed jobs (prefer already-ingested documents across multiple schemas)
|
||||
3. [ ] Hand-annotate 50-100 concepts per document — primary annotation by Atlas, 20% subset verified by a second annotator
|
||||
4. [ ] Build comparison script: `scripts/validate_extraction.py` that takes extractor output and calibration data, produces confusion matrix and per-type metrics
|
||||
5. [ ] Run the *current* extractor against the calibration dataset — no changes to config, no new prompts, just measure what we have
|
||||
6. [ ] Record baseline metrics in `validation/baseline_metrics.md`
|
||||
|
||||
**Deliverable:** Calibration dataset, validation script, baseline metrics. These are the yardstick every subsequent change is measured against.
|
||||
|
||||
**Estimated effort:** Medium — annotation labor is the bottleneck (150-500 concepts). The script is straightforward.
|
||||
|
||||
**Why this comes first:** Without a baseline, Phase 1 changes are invisible — we won't know if they helped or hurt. Running baseline first means every subsequent phase has a number to beat.
|
||||
|
||||
### Phase 1 — Enhanced Type Specs & Prompt Engineering
|
||||
|
||||
**Goal:** Upgrade the type specification format, add decision trees, rename one type, assemble prompts properly. The biggest accuracy gains per hour of work.
|
||||
|
||||
**Tasks:**
|
||||
1. [ ] Add `scope` field to each of the 7 schemas in `config.yaml`
|
||||
@@ -510,71 +676,103 @@ This is not a one-time exercise. As new source types are added or existing schem
|
||||
|
||||
**Estimated effort:** Small — mostly YAML editing and prompt engineering.
|
||||
|
||||
### Phase 2 — Calibration (validation infrastructure)
|
||||
### Phase 2 — Section-Aware Extraction
|
||||
|
||||
**Goal:** Build the measurement infrastructure to validate that Phase 1 changes improve extraction quality.
|
||||
**Goal:** Pass the structural context the pipeline already produces into the extractor prompt, so the LLM can use position, chapter titles, and section headings to inform type selection.
|
||||
|
||||
**Tasks:**
|
||||
1. [ ] Create `validation/calibration/` directory structure
|
||||
2. [ ] Hand-annotate calibration dataset (3-5 documents, 50-100 concepts each) — primary annotation by Atlas, 20% subset verified by a second annotator
|
||||
3. [ ] Build comparison script: `scripts/validate_extraction.py` that takes extractor output and calibration data, produces confusion matrix and per-type metrics
|
||||
4. [ ] Run baseline measurement with current extractor against calibration dataset (before Phase 1 changes, if possible — otherwise against Phase 1 output)
|
||||
5. [ ] Document baseline metrics in `validation/baseline_metrics.md`
|
||||
1. [ ] Identify where in the pipeline the extraction prompt is assembled
|
||||
2. [ ] Wire section metadata (chapter title, section heading, position fraction) from `structure.json` and summaries into the prompt builder
|
||||
3. [ ] Add the section context block to the extraction prompt template (see §4.6 for format)
|
||||
4. [ ] Test on 1-2 existing documents to verify the context appears correctly
|
||||
5. [ ] Update `docs/specs/concept-taxonomy-reference.md` to document the new prompt structure
|
||||
|
||||
**Deliverable:** Calibration dataset, validation script, baseline metrics.
|
||||
**Deliverable:** Section-aware extraction prompts using existing pipeline metadata.
|
||||
|
||||
**Estimated effort:** Medium — annotation is the bottleneck.
|
||||
**Estimated effort:** Small-medium — the data exists, this is wiring and prompt adjustment.
|
||||
|
||||
### Phase 3 — Iterate (tuning cycle)
|
||||
### Phase 3 — Re-Measure & Iterate
|
||||
|
||||
**Goal:** Use calibration data to tune type descriptions until ≥85% agreement is achieved.
|
||||
**Goal:** Run the calibrated system against the same baseline dataset. Compare to Phase 0 numbers. Tune until ≥85% agreement.
|
||||
|
||||
**Tasks:**
|
||||
1. [ ] Run extractor with Phase 1 type specs against calibration dataset
|
||||
2. [ ] Generate confusion matrix
|
||||
3. [ ] For each confused pair, either: (a) sharpen descriptions with more specific disambiguation rules, or (b) if confusion persists across iterations, consider merging the types
|
||||
4. [ ] Re-run and re-measure
|
||||
5. [ ] When stable, finalize descriptions in `config.yaml`
|
||||
6. [ ] Re-index any existing ingested concepts that were extracted with the old type specs (optional — may not be worth it if changes are description-only)
|
||||
1. [ ] Run extractor with Phase 1+2 changes against the calibration dataset
|
||||
2. [ ] Compare results to Phase 0 baseline — which types improved, which got worse, which are unchanged
|
||||
3. [ ] Generate confusion matrix
|
||||
4. [ ] For each confused pair, either: (a) sharpen descriptions with more specific disambiguation rules, or (b) if confusion persists across iterations, consider merging the types
|
||||
5. [ ] Re-run and re-measure
|
||||
6. [ ] When stable, finalize descriptions in `config.yaml`
|
||||
7. [ ] Record final metrics in `validation/final_metrics.md`
|
||||
8. [ ] Re-index any existing ingested concepts that were extracted with the old type specs (optional — may not be worth it if changes are description-only and the total data is small)
|
||||
|
||||
**Deliverable:** Finalized type specs with measured quality metrics.
|
||||
**Deliverable:** Finalized type specs with measured quality metrics. Before/after comparison.
|
||||
|
||||
**Estimated effort:** Medium — depends on how many iterations are needed.
|
||||
|
||||
### Phase 4 — Extend (optional, post-validation)
|
||||
### Phase 4 — Coherence Relationship Redesign
|
||||
|
||||
**Goal:** Apply validated patterns to new source types or schema extensions.
|
||||
**Goal:** Replace the flat 7-type coherence vocabulary with the dimensional classification system (§4.7), enabling richer relationship expression and better graph reasoning.
|
||||
|
||||
**Tasks:**
|
||||
1. [ ] Define the coherence dimensional schema in `config.yaml` (alignment, function, scope)
|
||||
2. [ ] Define the mapping from dimensional combinations to primary labels (for backward compatibility)
|
||||
3. [ ] Update the coherence extraction prompt to classify along each dimension independently
|
||||
4. [ ] Update the coherence storage schema in `graph.db` to store dimensional scores alongside the derived label
|
||||
5. [ ] Re-classify existing coherence relationships by running the new prompt against stored concept pairs (batch operation)
|
||||
6. [ ] Update any downstream code that reads coherence relationships to handle the new format
|
||||
7. [ ] Update `docs/specs/concept-taxonomy-reference.md` to document the new coherence types
|
||||
|
||||
**Deliverable:** Dimensional coherence classification with backward-compatible labels, migrated existing data.
|
||||
|
||||
**Estimated effort:** Medium — the prompt and schema changes are straightforward; the re-classification batch job is the main work.
|
||||
|
||||
**Why this comes after Phase 3:** The coherence layer depends on concept types being stable. If we change the concept taxonomy and the coherence vocabulary simultaneously, we can't tell which change caused any observed improvement or regression.
|
||||
|
||||
### Phase 5 — Extend (optional, post-validation)
|
||||
|
||||
**Goal:** Apply validated patterns to new source types, add diagnostic infrastructure if needed, explore advanced features.
|
||||
|
||||
**Tasks:**
|
||||
1. [ ] If adding new source types, use the enhanced type specification format from Phase 1
|
||||
2. [ ] Add calibration entries for new source types before deploying
|
||||
3. [ ] Consider adding `source_position`-based extraction rules (e.g., in `academic_paper`, weight `hypothesis` higher in introduction sections, `finding` higher in results sections) — now that we have calibration data to measure whether position-awareness improves accuracy
|
||||
4. [ ] If calibration revealed systematic confusion that needs deeper diagnosis, add `extraction_rationale` and `adjacent_type_score` as optional metadata fields — but only if there's a specific problem to solve
|
||||
3. [ ] If calibration revealed systematic confusion that needs deeper diagnosis, add `extraction_rationale` and `adjacent_type_score` as optional metadata fields — but only if there's a specific problem to solve
|
||||
4. [ ] Consider adding `source_position`-based extraction rules (e.g., in `academic_paper`, weight `hypothesis` higher in introduction sections, `finding` higher in results sections) — now that we have calibration data to measure whether position-awareness improves accuracy
|
||||
5. [ ] Build a coherence graph visualization or analysis tool (optional — would help with qualitative review of relationship quality)
|
||||
|
||||
**Deliverable:** Extension-ready architecture.
|
||||
**Deliverable:** Extension-ready architecture with ongoing measurement.
|
||||
|
||||
---
|
||||
|
||||
## 7. Open Questions
|
||||
|
||||
These are decisions that should be made before or during Phase 1, but don't block starting the work:
|
||||
These are decisions that should be made before or during Phase 1, but don't block starting Phase 0:
|
||||
|
||||
1. **Should we add `source_position` as a required extraction field?** Currently proposed as optional in Phase 1. Making it required means every extraction includes section-aware position, which helps with hypothesis/finding disambiguation in academic papers. But it depends on the parsed document providing section labels consistently. Recommendation: start optional, require it if Phase 2 calibration shows position would have resolved confusions.
|
||||
1. **Should we add `source_position` as a required extraction field?** Currently proposed as optional in Phase 1. Making it required means every extraction includes section-aware position, which helps with hypothesis/finding disambiguation in academic papers. But it depends on the parsed document providing section labels consistently. Recommendation: start optional, require it if Phase 3 calibration shows position would have resolved confusions.
|
||||
|
||||
2. **What's the threshold for creating a new schema vs. extending an existing one?** If we ingest a new document type (e.g., `podcast_transcript`), should it get its own schema or use the closest existing one? Recommendation: start by mapping to the closest existing schema, and only create a new schema if calibration shows ≥30% of concepts don't fit any existing type cleanly.
|
||||
|
||||
3. **Should the coherence dimensional scores be stored as separate fields or as a structured object?** Separate fields are simpler to query in SQLite. A structured JSON object is more flexible. Recommendation: separate fields for the three dimensions (alignment, function, scope) plus the derived label, since these are small and enumerable.
|
||||
|
||||
4. **What's the priority order between Phase 2 (section-aware) and Phase 4 (coherence)?** Phase 2 is smaller and directly supports Phase 3 measurement. Phase 4 is independent and can run in parallel if resources allow. Recommendation: sequential (Phase 2 → Phase 3 → Phase 4) unless you have two developers.
|
||||
|
||||
---
|
||||
|
||||
## 8. What This Plan Does NOT Change
|
||||
|
||||
To be explicit about stability:
|
||||
|
||||
- **Pipeline architecture** — parse → structure → summarize → classify → extract → index → coherence is preserved. We are changing what goes *into* the prompts, not how the pipeline flows.
|
||||
- **7 source types** — keeping all 7 (book, academic_paper, article, technical_document, instruction_manual, legal_document, fiction)
|
||||
- **7 schemas** — keeping the one-schema-per-source-type architecture
|
||||
- **7 coherence relationship types** — not touched by this plan
|
||||
- **Pipeline architecture** — parse → structure → summarize → classify → extract → index → coherence is preserved
|
||||
- **Chroma collections** — `ingested_concepts`, `knowledge_base`, `agent_memory` structure preserved (Gitea #38 may address this separately)
|
||||
- **7 schemas** — keeping the one-schema-per-source-type architecture (confirmed by Semantic Arts' modular design principle)
|
||||
- **Type counts** — staying at 6-7 types per schema (one rename, no additions or removals in Phase 1)
|
||||
- **Chroma collections** — `ingested_concepts`, `knowledge_base`, `agent_memory` structure preserved (Gitea #38 may address this separately)
|
||||
- **Classifier → Extractor split** — the two-pass LLM architecture (classify schema, then extract concepts) is correct and stays
|
||||
|
||||
What *does* change:
|
||||
- **Prompt content and structure** — richer type specs, decision trees, section context
|
||||
- **Config format** — from flat type labels to structured type specifications
|
||||
- **Coherence vocabulary** — from flat 7-type list to dimensional classification
|
||||
- **Measurement** — from "we don't know" to a baseline number and a target
|
||||
|
||||
---
|
||||
|
||||
@@ -582,12 +780,13 @@ To be explicit about stability:
|
||||
|
||||
- Rosch, E. (1978). *Principles of Categorization.* — Basic-level categories, prototypicality, cognitive economy
|
||||
- Gozli, D. (2023). *On the Relative Nature of Categories.* — Sense vs. choice categories, deviations from basic-level
|
||||
- Mammen, J. — Sense vs. choice categories (as discussed in Gozli 2023)
|
||||
- NNGroup / Taxon.dev — MECE taxonomy design, 10-step process, consistent labels
|
||||
- Semantic Arts — 8 ontology design schools, modular design, scope definition
|
||||
- Visser, S. et al. (2021). *Argument Scheme Annotation.* — Inter-annotator agreement, Walton's 60 schemes, Wagemans' Periodic Table, decision tree heuristics
|
||||
- LandingAI (2025). *LLM Extraction Schema Design.* — Descriptive field names, document-mirroring, data type specificity
|
||||
- Mammen, J. — Sense vs. choice categories (as discussed in Gozli 2023)
|
||||
- Visser, S. et al. (2021). *Argument Scheme Annotation.* — Inter-annotator agreement, Walton's 60 schemes, Wagemans' Periodic Table, decision tree heuristics, content vs. function distinction
|
||||
- Walton, D. — 60 argument schemes (empirically-motivated classification)
|
||||
- Wagemans, J. — Periodic Table of Arguments (36 types, factorial typology)
|
||||
- LandingAI (2025). *LLM Extraction Schema Design.* — Descriptive field names, document-mirroring, data type specificity
|
||||
|
||||
---
|
||||
|
||||
@@ -595,17 +794,21 @@ To be explicit about stability:
|
||||
|
||||
| What | Current | Proposed | Phase |
|
||||
|---|---|---|---|
|
||||
| **Measurement** | None | Baseline calibration dataset + comparison script + metrics | 0 |
|
||||
| `book.contradiction` | `contradiction` | Rename to `counterargument` | 1 |
|
||||
| `instruction_manual` default | `tip` | Change to `step` | 1 |
|
||||
| Type descriptions | 1-sentence label | Structured spec (long desc + cues + disambiguation + examples + priority) | 1 |
|
||||
| Schema scope | Implicit (schema name) | Explicit `scope` field | 1 |
|
||||
| Extractor prompt | Type list only | Type list + binary decision tree for ambiguous schemas | 1 |
|
||||
| Prompt assembly | Not specified | Explicit field ordering and token budget guidance (§4.4) | 1 |
|
||||
| Prompt assembly | Not specified | Explicit field ordering, token budget, example format (§4.4) | 1 |
|
||||
| Concept metadata | Current fields | Add `source_position` (optional) | 1 |
|
||||
| Legacy data migration | Not addressed | Bulk-update `contradiction` → `counterargument` in Chroma | 1 |
|
||||
| Validation infrastructure | None | Calibration dataset + comparison script + baseline metrics | 2 |
|
||||
| **Section context** | Not passed to extractor | Chapter title, section heading, position fraction in prompt | 2 |
|
||||
| **Re-measurement** | N/A | Run against calibration dataset, compare to baseline, iterate to ≥85% | 3 |
|
||||
| **Coherence relationships** | Flat 7-type list | Dimensional classification (alignment × function × scope) with derived labels | 4 |
|
||||
| **Coherence migration** | N/A | Re-classify existing relationships via batch prompt | 4 |
|
||||
| Quality target | Not measured | ≥85% type agreement vs. hand labels | 3 |
|
||||
| Diagnostic metadata | Not applicable | `extraction_rationale`, `adjacent_type_score` — only if calibration reveals need | 4 |
|
||||
| Diagnostic metadata | Not applicable | `extraction_rationale`, `adjacent_type_score` — only if calibration reveals need | 5 |
|
||||
|
||||
## Appendix B: Example Enhanced Type Spec — `book.claim`
|
||||
|
||||
@@ -636,3 +839,63 @@ claim:
|
||||
- "Expert intuition is more reliable than we think, but only in environments with regular feedback."
|
||||
priority: 1
|
||||
```
|
||||
|
||||
## Appendix C: Example Coherence Dimensional Classification
|
||||
|
||||
**Old format:**
|
||||
```json
|
||||
{
|
||||
"source_concept": "concept_abc123",
|
||||
"target_concept": "concept_def456",
|
||||
"relationship": "agrees"
|
||||
}
|
||||
```
|
||||
|
||||
**New format:**
|
||||
```json
|
||||
{
|
||||
"source_concept": "concept_abc123",
|
||||
"target_concept": "concept_def456",
|
||||
"alignment": "supportive",
|
||||
"function": "warrants",
|
||||
"scope": "same",
|
||||
"derived_label": "supports",
|
||||
"confidence": 0.82
|
||||
}
|
||||
```
|
||||
|
||||
**Derived label rules (complete):**
|
||||
|
||||
| alignment | function | scope | derived_label |
|
||||
|---|---|---|---|
|
||||
| supportive | corroborates | same | `same_idea` |
|
||||
| supportive | corroborates | broader/narrower | `agrees` |
|
||||
| supportive | warrants | any | `supports` |
|
||||
| supportive | exemplifies | narrower | `exemplifies` |
|
||||
| supportive | specializes | narrower | `specializes` |
|
||||
| supportive | extends | broader | `extends` |
|
||||
| opposing | rebuts | any | `contradicts` |
|
||||
| opposing | undermines | any | `undermines` |
|
||||
| opposing | qualifies | any | `qualifies` |
|
||||
| neutral | extends | broader | `extends` |
|
||||
| neutral | specializes | narrower | `specializes` |
|
||||
|
||||
## Appendix D: Phase Dependency Graph
|
||||
|
||||
```
|
||||
Phase 0 (Baseline)
|
||||
↓
|
||||
Phase 1 (Type Specs + Prompts)
|
||||
↓
|
||||
Phase 2 (Section-Aware Extraction)
|
||||
↓
|
||||
Phase 3 (Re-Measure + Iterate) ← feeds back to Phase 1 if needed
|
||||
↓
|
||||
Phase 4 (Coherence Redesign) ← independent of Phase 2, could parallel
|
||||
↓
|
||||
Phase 5 (Extend) ← only if needed
|
||||
```
|
||||
|
||||
**Critical path:** Phase 0 → Phase 1 → Phase 2 → Phase 3. These must be sequential because each depends on the previous.
|
||||
|
||||
**Parallelizable:** Phase 4 (coherence) is independent of Phases 2-3 and could run in parallel with a separate developer if resources allow. The only constraint is that concept types should be stable (Phase 3 complete or near-complete) before finalizing coherence, since coherence relationships reference concept types.
|
||||
Reference in New Issue
Block a user