Files
VuongKB/docs/specs/concept-taxonomy-reference.md
T

8.5 KiB

created, id, keywords, links, modified, summary, tags, title
created id keywords links modified summary tags title
2026-04-14T20:34:12+00:00 0874ad6c-7892-460c-8031-2b37187d6944
saga
ingestion
taxonomy
concept-schema
concept-type
coherence
chroma
source-type
gitea-37
gitea-38
2026-04-14T20:34:12+00:00 Complete reference for Saga's ingestion pipeline concept taxonomy — source types, concept schemas, concept types, coherence relationships, and pipeline structure. Baseline for Gitea #37/#38 revision work.
spec
saga
ingestion
taxonomy
reference
Concept Taxonomy Reference — Saga Ingestion Pipeline

Concept Taxonomy Reference

Source: agents/saga/config.yaml Updated: 2026-04-14 Status: Current baseline — under revision (Gitea #37, #38)

Complete reference for Saga's ingestion pipeline — source types, concept schemas, concept types, and coherence relationships.


Source Types (7)

What Saga classifies a document as before extraction:

Source Type Description
book Long-form non-fiction: popular science, history, philosophy, biography, self-help, business
academic_paper Formal research paper with abstract, methodology, results, and citations
article Short-form argument or analysis: essay, opinion piece, news article, blog post, review
technical_document Software documentation, API reference, technical specification, or system design document
instruction_manual How-to guide, user manual, tutorial, or procedural reference
legal_document Contract, policy, regulation, legal brief, or compliance document
fiction Narrative work of imagination: novel, short story, screenplay, literary fiction

Concept Schemas (7, one per source type)

Each schema defines what concept types the extractor looks for. The classifier LLM selects a schema from the master summary; the extractor then uses only those types in its prompt and validates output against them.

book (default type: claim)

Type Description
claim A stated argument or position the author defends
definition A precise explanation of a term, model, or idea
framework A structured model or system of thinking the author proposes
principle A general rule or guideline the author advocates
example A concrete case or story illustrating a broader point
contradiction A claim that challenges conventional wisdom or another view
question An open question or puzzle the author raises

academic_paper (default type: finding)

Type Description
hypothesis A proposition being tested or argued
finding An empirical result or observed outcome
method A technique, procedure, or experimental design used
limitation A constraint, caveat, or boundary condition the authors acknowledge
contribution A claimed advance or novelty over prior work
definition A formal definition of a term or construct
example A concrete illustration, dataset, or case study

article (default type: claim)

Type Description
claim The central assertion or argument being made
evidence A fact, statistic, or source cited to support a claim
counterargument A view the author acknowledges and rebuts
definition A term scoped or defined for the argument
example A concrete illustration of a point
question An open question the author raises or leaves unresolved

technical_document (default type: definition)

Type Description
definition A term, component, or concept precisely defined
requirement A stated constraint, rule, or system requirement
procedure A sequence of steps or process description
parameter A configurable value, option, or input specification
warning A caution, known failure mode, or deprecation notice
example A concrete usage or code example

instruction_manual (default type: tip)

Type Description
step A discrete action in a procedure
prerequisite A condition or setup required before proceeding
warning A safety caution or common mistake to avoid
tip A recommended shortcut or best practice
troubleshooting A known problem and its resolution
example A concrete illustration of correct usage
Type Description
obligation A duty or required action imposed on a party
right An entitlement or permission granted to a party
condition A circumstance that triggers or modifies an obligation or right
exception A carve-out or exclusion from a rule
definition A term given a specific legal meaning
penalty A consequence or remedy for non-compliance

fiction (default type: theme)

Type Description
character_trait A defining quality, motivation, or behaviour pattern of a character
plot_event A significant event that advances or changes the narrative
theme A recurring idea, moral, or philosophical thread in the work
symbol An object, place, or event that carries deeper meaning
relationship A dynamic or tension between two or more characters
setting A described place, time period, or atmosphere

Pipeline Structure

source file
  -> parse (epub/pdf/html/markdown)
  -> structure (chapters/sections with token budgets)
  -> summarize (bottom-up: sections -> chapters -> master)
  -> classify (select concept schema from master summary)
  -> extract concepts (per-section, schema-driven types)
  -> index (KB notes to Atlas, concepts to Chroma)
  -> coherence (cross-document relationship graph)

Phase details

Phase What it does Output
structure Parse documents, break into chapters and sections structure.json
summarize Bottom-up map-reduce summarization summaries/sections/*.md, summaries/chapters/*.md, summaries/master.md
classify LLM selects concept schema from master summary Schema name in manifest
concepts Per-section LLM extraction with schema-typed output concepts/*.json
indexing Write KB notes + upsert concepts to Chroma staging/kb_notes/*.md, ingested_concepts collection
coherence Cross-document relationship detection via embedding + LLM coherence/graph.db (SQLite)

Coherence Relationship Types (7)

Post-extraction, cross-document relationships between concepts:

Relationship Description
same_idea Two concepts express the same underlying idea
agrees One concept supports or corroborates another
contradicts One concept opposes or refutes another
extends One concept builds upon or elaborates another
specializes One concept is a specific case of a more general one
exemplifies One concept provides a concrete example of another
undermines One concept weakens the basis for another without directly contradicting it

Chroma Collections

Collection Content Queried by
knowledge_base Hand-written KB notes (chunked, embedded) Atlas search_notes
ingested_concepts Saga concept extractions with metadata Atlas search_ingested
agent_memory Per-agent LTM (shared collection, agent_id filter) Memory tools

Concept metadata fields (ingested_concepts)

Each concept in the ingested_concepts collection carries:

Field Description
job_id Saga job identifier
title Source document title
author Source document author
concept_schema Which schema was used (e.g. book, fiction)
concept_type The specific type (e.g. claim, character_trait)
confidence Extraction confidence score
terms Key terms associated with the concept
chapter_id Which chapter the concept came from
section_id Which section within the chapter
verbatim Whether the text is a direct quote

Ingested Jobs (current)

Job Title Schema KB Notes Source
163a58dc Beautiful Website UI/UX Design book 5 chapters web corpus
24e3e147 OCC OCEAN PAD affective models book 1 chapter web corpus
7cdc3792 Thinking, Fast and Slow book 49 chapters epub
b59c147c Dungeon Crawler Carl fiction 55 chapters epub

  • Config: agents/saga/config.yaml (source of truth for schemas)
  • Taxonomy refinement: Gitea #37
  • Collection architecture redesign: Gitea #38
  • Saga spec: docs/saga-spec.md