From d7f7515f22e3039674d492d5310c8a4061b29828 Mon Sep 17 00:00:00 2001 From: Atlas Date: Tue, 14 Apr 2026 13:34:12 -0700 Subject: [PATCH] Create: docs/specs/concept-taxonomy-reference.md --- docs/specs/concept-taxonomy-reference.md | 225 +++++++++++++++++++++++ 1 file changed, 225 insertions(+) create mode 100644 docs/specs/concept-taxonomy-reference.md diff --git a/docs/specs/concept-taxonomy-reference.md b/docs/specs/concept-taxonomy-reference.md new file mode 100644 index 0000000..18b2308 --- /dev/null +++ b/docs/specs/concept-taxonomy-reference.md @@ -0,0 +1,225 @@ +--- +created: '2026-04-14T20:34:12+00:00' +id: 0874ad6c-7892-460c-8031-2b37187d6944 +keywords: +- saga +- ingestion +- taxonomy +- concept-schema +- concept-type +- coherence +- chroma +- source-type +- gitea-37 +- gitea-38 +links: [] +modified: '2026-04-14T20:34:12+00:00' +summary: 'Complete reference for Saga''s ingestion pipeline concept taxonomy — source + types, concept schemas, concept types, coherence relationships, and pipeline structure. + Baseline for Gitea #37/#38 revision work.' +tags: +- spec +- saga +- ingestion +- taxonomy +- reference +title: Concept Taxonomy Reference — Saga Ingestion Pipeline +--- + +# Concept Taxonomy Reference + +**Source:** `agents/saga/config.yaml` +**Updated:** 2026-04-14 +**Status:** Current baseline — under revision (Gitea #37, #38) + +Complete reference for Saga's ingestion pipeline — source types, concept schemas, concept types, and coherence relationships. + +--- + +## Source Types (7) + +What Saga classifies a document as before extraction: + +| Source Type | Description | +|---|---| +| `book` | Long-form non-fiction: popular science, history, philosophy, biography, self-help, business | +| `academic_paper` | Formal research paper with abstract, methodology, results, and citations | +| `article` | Short-form argument or analysis: essay, opinion piece, news article, blog post, review | +| `technical_document` | Software documentation, API reference, technical specification, or system design document | +| `instruction_manual` | How-to guide, user manual, tutorial, or procedural reference | +| `legal_document` | Contract, policy, regulation, legal brief, or compliance document | +| `fiction` | Narrative work of imagination: novel, short story, screenplay, literary fiction | + +--- + +## Concept Schemas (7, one per source type) + +Each schema defines what concept types the extractor looks for. The classifier LLM selects a schema from the master summary; the extractor then uses only those types in its prompt and validates output against them. + +### `book` (default type: `claim`) + +| Type | Description | +|---|---| +| `claim` | A stated argument or position the author defends | +| `definition` | A precise explanation of a term, model, or idea | +| `framework` | A structured model or system of thinking the author proposes | +| `principle` | A general rule or guideline the author advocates | +| `example` | A concrete case or story illustrating a broader point | +| `contradiction` | A claim that challenges conventional wisdom or another view | +| `question` | An open question or puzzle the author raises | + +### `academic_paper` (default type: `finding`) + +| Type | Description | +|---|---| +| `hypothesis` | A proposition being tested or argued | +| `finding` | An empirical result or observed outcome | +| `method` | A technique, procedure, or experimental design used | +| `limitation` | A constraint, caveat, or boundary condition the authors acknowledge | +| `contribution` | A claimed advance or novelty over prior work | +| `definition` | A formal definition of a term or construct | +| `example` | A concrete illustration, dataset, or case study | + +### `article` (default type: `claim`) + +| Type | Description | +|---|---| +| `claim` | The central assertion or argument being made | +| `evidence` | A fact, statistic, or source cited to support a claim | +| `counterargument` | A view the author acknowledges and rebuts | +| `definition` | A term scoped or defined for the argument | +| `example` | A concrete illustration of a point | +| `question` | An open question the author raises or leaves unresolved | + +### `technical_document` (default type: `definition`) + +| Type | Description | +|---|---| +| `definition` | A term, component, or concept precisely defined | +| `requirement` | A stated constraint, rule, or system requirement | +| `procedure` | A sequence of steps or process description | +| `parameter` | A configurable value, option, or input specification | +| `warning` | A caution, known failure mode, or deprecation notice | +| `example` | A concrete usage or code example | + +### `instruction_manual` (default type: `tip`) + +| Type | Description | +|---|---| +| `step` | A discrete action in a procedure | +| `prerequisite` | A condition or setup required before proceeding | +| `warning` | A safety caution or common mistake to avoid | +| `tip` | A recommended shortcut or best practice | +| `troubleshooting` | A known problem and its resolution | +| `example` | A concrete illustration of correct usage | + +### `legal_document` (default type: `obligation`) + +| Type | Description | +|---|---| +| `obligation` | A duty or required action imposed on a party | +| `right` | An entitlement or permission granted to a party | +| `condition` | A circumstance that triggers or modifies an obligation or right | +| `exception` | A carve-out or exclusion from a rule | +| `definition` | A term given a specific legal meaning | +| `penalty` | A consequence or remedy for non-compliance | + +### `fiction` (default type: `theme`) + +| Type | Description | +|---|---| +| `character_trait` | A defining quality, motivation, or behaviour pattern of a character | +| `plot_event` | A significant event that advances or changes the narrative | +| `theme` | A recurring idea, moral, or philosophical thread in the work | +| `symbol` | An object, place, or event that carries deeper meaning | +| `relationship` | A dynamic or tension between two or more characters | +| `setting` | A described place, time period, or atmosphere | + +--- + +## Pipeline Structure + +``` +source file + -> parse (epub/pdf/html/markdown) + -> structure (chapters/sections with token budgets) + -> summarize (bottom-up: sections -> chapters -> master) + -> classify (select concept schema from master summary) + -> extract concepts (per-section, schema-driven types) + -> index (KB notes to Atlas, concepts to Chroma) + -> coherence (cross-document relationship graph) +``` + +### Phase details + +| Phase | What it does | Output | +|---|---|---| +| `structure` | Parse documents, break into chapters and sections | `structure.json` | +| `summarize` | Bottom-up map-reduce summarization | `summaries/sections/*.md`, `summaries/chapters/*.md`, `summaries/master.md` | +| `classify` | LLM selects concept schema from master summary | Schema name in manifest | +| `concepts` | Per-section LLM extraction with schema-typed output | `concepts/*.json` | +| `indexing` | Write KB notes + upsert concepts to Chroma | `staging/kb_notes/*.md`, `ingested_concepts` collection | +| `coherence` | Cross-document relationship detection via embedding + LLM | `coherence/graph.db` (SQLite) | + +--- + +## Coherence Relationship Types (7) + +Post-extraction, cross-document relationships between concepts: + +| Relationship | Description | +|---|---| +| `same_idea` | Two concepts express the same underlying idea | +| `agrees` | One concept supports or corroborates another | +| `contradicts` | One concept opposes or refutes another | +| `extends` | One concept builds upon or elaborates another | +| `specializes` | One concept is a specific case of a more general one | +| `exemplifies` | One concept provides a concrete example of another | +| `undermines` | One concept weakens the basis for another without directly contradicting it | + +--- + +## Chroma Collections + +| Collection | Content | Queried by | +|---|---|---| +| `knowledge_base` | Hand-written KB notes (chunked, embedded) | Atlas `search_notes` | +| `ingested_concepts` | Saga concept extractions with metadata | Atlas `search_ingested` | +| `agent_memory` | Per-agent LTM (shared collection, agent_id filter) | Memory tools | + +### Concept metadata fields (ingested_concepts) + +Each concept in the `ingested_concepts` collection carries: + +| Field | Description | +|---|---| +| `job_id` | Saga job identifier | +| `title` | Source document title | +| `author` | Source document author | +| `concept_schema` | Which schema was used (e.g. `book`, `fiction`) | +| `concept_type` | The specific type (e.g. `claim`, `character_trait`) | +| `confidence` | Extraction confidence score | +| `terms` | Key terms associated with the concept | +| `chapter_id` | Which chapter the concept came from | +| `section_id` | Which section within the chapter | +| `verbatim` | Whether the text is a direct quote | + +--- + +## Ingested Jobs (current) + +| Job | Title | Schema | KB Notes | Source | +|---|---|---|---|---| +| `163a58dc` | Beautiful Website UI/UX Design | book | 5 chapters | web corpus | +| `24e3e147` | OCC OCEAN PAD affective models | book | 1 chapter | web corpus | +| `7cdc3792` | Thinking, Fast and Slow | book | 49 chapters | epub | +| `b59c147c` | Dungeon Crawler Carl | fiction | 55 chapters | epub | + +--- + +## Related + +- Config: `agents/saga/config.yaml` (source of truth for schemas) +- Taxonomy refinement: Gitea #37 +- Collection architecture redesign: Gitea #38 +- Saga spec: `docs/saga-spec.md` \ No newline at end of file