Ingestion

What the word means

“Ingestion” is borrowed from data engineering. Generically it means “pulling data into your system from somewhere else.” In RAG specifically, it means:

Source documents → chunks → embeddings → stored in a vector database.

Once ingestion is done, your system can answer questions about those documents. Without ingestion, the system has nothing to retrieve from.

What ingestion does in my RAG app

In my RAG app, src/rag_app/ingest.py is the entire ingestion script. It runs once (or whenever my notes change). What it does, in order:

1. Load - walks my notes folder, reads every .md file into memory as a LangChain Document object (text + metadata like the file path)
2. Chunk - splits each document into ~1000-character pieces. A document might be 10,000 chars; ingestion breaks it into ~10 chunks. Why: LLMs have context windows, retrieval is more precise on smaller units, and you want the model to see relevant paragraphs, not entire essays
3. Embed - turns each chunk’s text into a vector (~1500 floating-point numbers) by calling OpenAI’s text-embedding-3-small API. This is the slow + costly step. Redis cache makes re-runs free for unchanged chunks
4. Persist - writes all the chunks + their vectors into chroma_db/chroma.sqlite3, along with metadata (which file each chunk came from)

After ingestion finishes, there’s a populated chroma_db/. That’s the index the query path reads from. Nothing more is needed until the notes change.

The split that matters: ingest vs query

This is the fundamental architecture split in any RAG system.

	Ingest	Query
When	Once, manually (when notes change)	Per user request
Speed	Slow (~15s for 100 chunks cold, ~1s warm)	Fast (~2s per question)
Cost	Embedding API + disk write	Embedding (1 call) + LLM (1 call)
Output	Populated chroma_db/	Answer + sources, returned to user
Code	ingest.py	rag.py + main.py

Ingest produces the index. Query consumes the index.

They’re completely separate processes. The FastAPI server doesn’t do ingestion; it only does queries against the already-built chroma_db/. That’s why if I add new notes, the running server doesn’t know about them until I re-ingest.

This is the same architectural pattern as a search engine: Google crawls and indexes the web (ingest), then users search against the index (query). Crawling is slow + happens offline; searching is fast + happens on every request.

The library analogy

Think of ingestion as preparing a library:

• You receive a stack of books (the notes)
• You catalog each book (read it, note what’s in it)
• You assign each book a “fingerprint” so you can find it later by topic (embedding)
• You shelve everything in an organized way (Chroma)

Then querying is a reader walking in:

• They ask for “books about ownership”
• The librarian looks up the topic in the index
• Pulls down the 4 most relevant books
• Hands them to the reader along with a summary

Ingestion is the librarian’s prep work. Querying is the reader’s request. The library never lets readers in until the prep is done.

Why this term shows up everywhere in RAG

Every RAG system has this same split. The variations are mostly in what gets ingested:

• My RAG app: markdown notes from a folder
• Glean: PDFs + Notion pages + Slack messages + Drive files + GitHub repos
• Harvey: court cases + contracts
• Cursor: code files from your repo
• NotebookLM: PDFs you upload

All of them have an “ingest pipeline” (read source → chunk → embed → store) and a “query pipeline” (embed question → retrieve → generate). The shape is identical; only the input format and the chunking strategy differ.

terminology aliases you’ll see in different codebases:

• ingest.py / ingestion service / indexer / loader pipeline / data pipeline - all mean the same thing
• query.py / retrieval service / inference / chat handler - the read side

Why chunking matters (a sub-concept)

Splitting documents into chunks isn’t arbitrary. Three reasons:

1. LLM context windows are finite. You can’t paste a 10,000-character document into a prompt every time. You retrieve the most relevant 2-3K characters and use those.
2. Retrieval precision improves on smaller units. A 10,000-character document might be 90% irrelevant to the question; retrieving just the relevant 1,000 characters gives the model a higher signal-to-noise ratio.
3. You need the chunk-to-file metadata for citations. When each chunk knows which file it came from, you can return source filenames in the response. Without the chunking layer, citations would be document-level only.

The trade-off: chunks too small = lose context that’s split across the boundary. Chunks too large = noise. 1000 characters with 200 overlap is a reasonable default for prose; you’d tune it differently for code (split by function), tables (split by row), or PDFs with figures.

What I noticed building my RAG app

• Ingestion is destructive on purpose in this project: shutil.rmtree(CHROMA_DIR) before rebuilding. Without that, re-runs would double-insert chunks. Destructive matches the mental model (you re-ingest because the corpus changed; you want the new state, not the union).
• The cache is on the embedding step, not the chunking step. Chunking is fast (pure Python text splitting). Embedding is the expensive bit.
• Adding new notes doesn’t change running queries until you re-ingest. The server holds an open handle to the old chroma_db/ via the lru_cache(maxsize=1) singleton. Restart the server or wait for the cache to refresh - in practice, a server restart is the cleaner choice.
• Each chunk keeps its file path in doc.metadata["source"] automatically. The citation system in rag.py reads that metadata to build the sources list in the response. The metadata that travels with chunks during ingest is what makes citations work at query time.