225 lines
8.5 KiB
Markdown
225 lines
8.5 KiB
Markdown
---
|
|
created: '2026-04-14T20:34:12+00:00'
|
|
id: 0874ad6c-7892-460c-8031-2b37187d6944
|
|
keywords:
|
|
- saga
|
|
- ingestion
|
|
- taxonomy
|
|
- concept-schema
|
|
- concept-type
|
|
- coherence
|
|
- chroma
|
|
- source-type
|
|
- gitea-37
|
|
- gitea-38
|
|
links: []
|
|
modified: '2026-04-14T20:34:12+00:00'
|
|
summary: 'Complete reference for Saga''s ingestion pipeline concept taxonomy — source
|
|
types, concept schemas, concept types, coherence relationships, and pipeline structure.
|
|
Baseline for Gitea #37/#38 revision work.'
|
|
tags:
|
|
- spec
|
|
- saga
|
|
- ingestion
|
|
- taxonomy
|
|
- reference
|
|
title: Concept Taxonomy Reference — Saga Ingestion Pipeline
|
|
---
|
|
|
|
# Concept Taxonomy Reference
|
|
|
|
**Source:** `agents/saga/config.yaml`
|
|
**Updated:** 2026-04-14
|
|
**Status:** Current baseline — under revision (Gitea #37, #38)
|
|
|
|
Complete reference for Saga's ingestion pipeline — source types, concept schemas, concept types, and coherence relationships.
|
|
|
|
---
|
|
|
|
## Source Types (7)
|
|
|
|
What Saga classifies a document as before extraction:
|
|
|
|
| Source Type | Description |
|
|
|---|---|
|
|
| `book` | Long-form non-fiction: popular science, history, philosophy, biography, self-help, business |
|
|
| `academic_paper` | Formal research paper with abstract, methodology, results, and citations |
|
|
| `article` | Short-form argument or analysis: essay, opinion piece, news article, blog post, review |
|
|
| `technical_document` | Software documentation, API reference, technical specification, or system design document |
|
|
| `instruction_manual` | How-to guide, user manual, tutorial, or procedural reference |
|
|
| `legal_document` | Contract, policy, regulation, legal brief, or compliance document |
|
|
| `fiction` | Narrative work of imagination: novel, short story, screenplay, literary fiction |
|
|
|
|
---
|
|
|
|
## Concept Schemas (7, one per source type)
|
|
|
|
Each schema defines what concept types the extractor looks for. The classifier LLM selects a schema from the master summary; the extractor then uses only those types in its prompt and validates output against them.
|
|
|
|
### `book` (default type: `claim`)
|
|
|
|
| Type | Description |
|
|
|---|---|
|
|
| `claim` | A stated argument or position the author defends |
|
|
| `definition` | A precise explanation of a term, model, or idea |
|
|
| `framework` | A structured model or system of thinking the author proposes |
|
|
| `principle` | A general rule or guideline the author advocates |
|
|
| `example` | A concrete case or story illustrating a broader point |
|
|
| `contradiction` | A claim that challenges conventional wisdom or another view |
|
|
| `question` | An open question or puzzle the author raises |
|
|
|
|
### `academic_paper` (default type: `finding`)
|
|
|
|
| Type | Description |
|
|
|---|---|
|
|
| `hypothesis` | A proposition being tested or argued |
|
|
| `finding` | An empirical result or observed outcome |
|
|
| `method` | A technique, procedure, or experimental design used |
|
|
| `limitation` | A constraint, caveat, or boundary condition the authors acknowledge |
|
|
| `contribution` | A claimed advance or novelty over prior work |
|
|
| `definition` | A formal definition of a term or construct |
|
|
| `example` | A concrete illustration, dataset, or case study |
|
|
|
|
### `article` (default type: `claim`)
|
|
|
|
| Type | Description |
|
|
|---|---|
|
|
| `claim` | The central assertion or argument being made |
|
|
| `evidence` | A fact, statistic, or source cited to support a claim |
|
|
| `counterargument` | A view the author acknowledges and rebuts |
|
|
| `definition` | A term scoped or defined for the argument |
|
|
| `example` | A concrete illustration of a point |
|
|
| `question` | An open question the author raises or leaves unresolved |
|
|
|
|
### `technical_document` (default type: `definition`)
|
|
|
|
| Type | Description |
|
|
|---|---|
|
|
| `definition` | A term, component, or concept precisely defined |
|
|
| `requirement` | A stated constraint, rule, or system requirement |
|
|
| `procedure` | A sequence of steps or process description |
|
|
| `parameter` | A configurable value, option, or input specification |
|
|
| `warning` | A caution, known failure mode, or deprecation notice |
|
|
| `example` | A concrete usage or code example |
|
|
|
|
### `instruction_manual` (default type: `tip`)
|
|
|
|
| Type | Description |
|
|
|---|---|
|
|
| `step` | A discrete action in a procedure |
|
|
| `prerequisite` | A condition or setup required before proceeding |
|
|
| `warning` | A safety caution or common mistake to avoid |
|
|
| `tip` | A recommended shortcut or best practice |
|
|
| `troubleshooting` | A known problem and its resolution |
|
|
| `example` | A concrete illustration of correct usage |
|
|
|
|
### `legal_document` (default type: `obligation`)
|
|
|
|
| Type | Description |
|
|
|---|---|
|
|
| `obligation` | A duty or required action imposed on a party |
|
|
| `right` | An entitlement or permission granted to a party |
|
|
| `condition` | A circumstance that triggers or modifies an obligation or right |
|
|
| `exception` | A carve-out or exclusion from a rule |
|
|
| `definition` | A term given a specific legal meaning |
|
|
| `penalty` | A consequence or remedy for non-compliance |
|
|
|
|
### `fiction` (default type: `theme`)
|
|
|
|
| Type | Description |
|
|
|---|---|
|
|
| `character_trait` | A defining quality, motivation, or behaviour pattern of a character |
|
|
| `plot_event` | A significant event that advances or changes the narrative |
|
|
| `theme` | A recurring idea, moral, or philosophical thread in the work |
|
|
| `symbol` | An object, place, or event that carries deeper meaning |
|
|
| `relationship` | A dynamic or tension between two or more characters |
|
|
| `setting` | A described place, time period, or atmosphere |
|
|
|
|
---
|
|
|
|
## Pipeline Structure
|
|
|
|
```
|
|
source file
|
|
-> parse (epub/pdf/html/markdown)
|
|
-> structure (chapters/sections with token budgets)
|
|
-> summarize (bottom-up: sections -> chapters -> master)
|
|
-> classify (select concept schema from master summary)
|
|
-> extract concepts (per-section, schema-driven types)
|
|
-> index (KB notes to Atlas, concepts to Chroma)
|
|
-> coherence (cross-document relationship graph)
|
|
```
|
|
|
|
### Phase details
|
|
|
|
| Phase | What it does | Output |
|
|
|---|---|---|
|
|
| `structure` | Parse documents, break into chapters and sections | `structure.json` |
|
|
| `summarize` | Bottom-up map-reduce summarization | `summaries/sections/*.md`, `summaries/chapters/*.md`, `summaries/master.md` |
|
|
| `classify` | LLM selects concept schema from master summary | Schema name in manifest |
|
|
| `concepts` | Per-section LLM extraction with schema-typed output | `concepts/*.json` |
|
|
| `indexing` | Write KB notes + upsert concepts to Chroma | `staging/kb_notes/*.md`, `ingested_concepts` collection |
|
|
| `coherence` | Cross-document relationship detection via embedding + LLM | `coherence/graph.db` (SQLite) |
|
|
|
|
---
|
|
|
|
## Coherence Relationship Types (7)
|
|
|
|
Post-extraction, cross-document relationships between concepts:
|
|
|
|
| Relationship | Description |
|
|
|---|---|
|
|
| `same_idea` | Two concepts express the same underlying idea |
|
|
| `agrees` | One concept supports or corroborates another |
|
|
| `contradicts` | One concept opposes or refutes another |
|
|
| `extends` | One concept builds upon or elaborates another |
|
|
| `specializes` | One concept is a specific case of a more general one |
|
|
| `exemplifies` | One concept provides a concrete example of another |
|
|
| `undermines` | One concept weakens the basis for another without directly contradicting it |
|
|
|
|
---
|
|
|
|
## Chroma Collections
|
|
|
|
| Collection | Content | Queried by |
|
|
|---|---|---|
|
|
| `knowledge_base` | Hand-written KB notes (chunked, embedded) | Atlas `search_notes` |
|
|
| `ingested_concepts` | Saga concept extractions with metadata | Atlas `search_ingested` |
|
|
| `agent_memory` | Per-agent LTM (shared collection, agent_id filter) | Memory tools |
|
|
|
|
### Concept metadata fields (ingested_concepts)
|
|
|
|
Each concept in the `ingested_concepts` collection carries:
|
|
|
|
| Field | Description |
|
|
|---|---|
|
|
| `job_id` | Saga job identifier |
|
|
| `title` | Source document title |
|
|
| `author` | Source document author |
|
|
| `concept_schema` | Which schema was used (e.g. `book`, `fiction`) |
|
|
| `concept_type` | The specific type (e.g. `claim`, `character_trait`) |
|
|
| `confidence` | Extraction confidence score |
|
|
| `terms` | Key terms associated with the concept |
|
|
| `chapter_id` | Which chapter the concept came from |
|
|
| `section_id` | Which section within the chapter |
|
|
| `verbatim` | Whether the text is a direct quote |
|
|
|
|
---
|
|
|
|
## Ingested Jobs (current)
|
|
|
|
| Job | Title | Schema | KB Notes | Source |
|
|
|---|---|---|---|---|
|
|
| `163a58dc` | Beautiful Website UI/UX Design | book | 5 chapters | web corpus |
|
|
| `24e3e147` | OCC OCEAN PAD affective models | book | 1 chapter | web corpus |
|
|
| `7cdc3792` | Thinking, Fast and Slow | book | 49 chapters | epub |
|
|
| `b59c147c` | Dungeon Crawler Carl | fiction | 55 chapters | epub |
|
|
|
|
---
|
|
|
|
## Related
|
|
|
|
- Config: `agents/saga/config.yaml` (source of truth for schemas)
|
|
- Taxonomy refinement: Gitea #37
|
|
- Collection architecture redesign: Gitea #38
|
|
- Saga spec: `docs/saga-spec.md` |