meowy

notes on what i build and what i learn

← Field map

Stack decisions for a RAG app

· note · budding · about wiki-rag

The starting point

The goal was a RAG service over a folder of markdown notes, with one binding constraint: the stack should mirror a real team’s production stack closely. That constraint did most of the architectural work. Without it, you have infinite freedom. With it, every choice gets one question: “does the team use it?”

The stack decisions, in order

These got locked in roughly in this sequence:

1. Python 3.13 + uv (language + toolchain)

Team uses Python 3.13. uv is the modern toolchain - one binary that replaces pip + virtualenv + pyenv + poetry. Picked it because it’s the only Python tooling that feels like cargo. One command for everything.

practical impact on the code: every command starts with uv run. Every dep is in pyproject.toml. There’s exactly one place to look.

2. FastAPI + Pydantic v2 (web framework + validation)

Team uses it. The alternative was Flask or starlette directly. FastAPI gives you typed routes, OpenAPI/Swagger for free, async-first design, and Pydantic validation at the boundary.

practical impact: every endpoint is @app.post("/query"). Every request body is a Pydantic class. You write zero manual validation - Pydantic does it from the type annotations.

3. LangChain 1.x (the RAG framework)

Team uses it. The alternative is “roll your own”: call OpenAI’s embeddings + write your own retriever + call OpenAI chat directly. That works but you’d reinvent chunking, vector store integrations, observability callbacks, eval adapters.

practical impact: you use LangChain’s Document shape (page_content + metadata), their OpenAIEmbeddings, their Chroma integration, their ChatOpenAI, their CacheBackedEmbeddings wrapper, their callback system. The trade-off is that LangChain is mid-migration in 1.x - some imports moved (from langchain_classic.embeddings instead of from langchain.embeddings), some APIs are deprecated. You pay for that with occasional shim files like _ragas_compat.py.

4. Chroma local file (the vector store)

Real options were Chroma local, pgvector (Postgres extension), or a managed store (Pinecone, Weaviate, Azure AI Search). Picked Chroma local because v1 is single-user on one machine.

practical impact: the vector store is just a SQLite file at chroma_db/chroma.sqlite3. No separate database server, no migrations, no users. The trade-off: single-writer, can’t scale to multiple replicas. The README’s “missing for prod” calls this out: pgvector is what you’d switch to before running two app instances.

5. Redis for the embedding cache

Embeddings are slow + cost money. Re-ingesting unchanged chunks shouldn’t pay that cost twice. Redis as a text-hash → vector key-value store solves this.

practical impact: in ingest.py and rag.py, the OpenAIEmbeddings instance gets wrapped with CacheBackedEmbeddings.from_bytes_store(underlying_embeddings=..., document_embedding_cache=store). The wrapper does the lookup-or-call logic automatically.

cost: Redis is a separate process. Docker became a hard prereq. Eventually wrapped in docker-compose.yml for persistence + auto-restart.

6. OpenAI for everything (gpt-4o-mini + text-embedding-3-small)

Considered splitting (Groq for inference to save cost, OpenAI for embeddings). Rejected the split because the team uses langchain-openai’s ChatOpenAI directly. Matching their stack exactly was the bigger signal than saving $0.50 in eval costs. One provider, one key, one client class.

practical impact: every call site is from langchain_openai import ChatOpenAI, OpenAIEmbeddings. Same provider for embeddings and chat. Total spend across the whole build was under $1.

7. LangFuse Cloud (observability)

Team uses it. Free tier covers personal use. v2 plan is to self-host via docker-compose to match the team’s deployment.

practical impact: one callback handler attached to LLM/chain invocations. Traces appear in LangFuse’s UI with token counts, cost, latency. LangFuse 4.x is OpenTelemetry-backed; the integration is one line of config={"callbacks": [handler]} on chain .ainvoke(...).

8. Ragas (eval)

Industry standard for RAG eval. Faithfulness + answer relevancy are the two metrics worth measuring early. Both work without ground-truth answers.

practical impact: eval.py loads a golden set, runs answer() on each question, builds a Ragas EvaluationDataset, scores with Faithfulness() and AnswerRelevancy(). Trade-off: Ragas 0.4 is mid-migration too, had to add a sys.modules stub for a removed langchain-community module.

What these decisions locked in

This is the part that matters: after these choices, your code patterns are basically determined.

  • All-OpenAI → from langchain_openai import ChatOpenAI, OpenAIEmbeddings everywhere
  • LangChain → use their Document shape, their embeddings interface, their vectorstore interface
  • FastAPI → async def handlers, Pydantic models, route decorators
  • Redis embedding cache → wrap your OpenAIEmbeddings once, use the wrapper everywhere
  • pydantic-settings → one Settings class, settings.openai_api_key, etc. throughout the code

You don’t have flexibility in how you call these things. The libraries expect specific shapes. The architecture decisions made the syntax choices for you.

The scaffold

After the decisions, the physical setup is fast:

uv init rag-app           # creates the folder + pyproject.toml + main.py + .python-version
cd rag-app
uv add fastapi langchain langchain-openai langchain-community chromadb redis langfuse pydantic-settings uvicorn
uv add --dev pytest ruff mypy ragas

Then .env.example gets committed with placeholder values, and .env (real secrets) goes in .gitignore. Standard Python pattern: templates committed, real secrets not.

The src/ layout decision came later in the build (I had to add [build-system] to pyproject.toml to make it work), but architecturally the choice is also part of phase 1: do you put code at the root or in src/<package>/? Picked src/ because the tooling (basedpyright, pytest) handles it cleaner.

The mental model

Architecture before code. Each library choice is a contract you sign with your future self. “I’m going to use Pydantic” means every request shape is a Pydantic class. “I’m going to use LangChain” means I use their abstractions instead of raw OpenAI calls. The hard work is in choosing well; the typing is mechanical after that.

Match the existing ecosystem. For Python RAG in 2026, the standard stack is: uv + FastAPI + LangChain + OpenAI + (some vector store) + (some observability). Picking the obvious option in each slot is a feature, not a bug - your code looks like everyone else’s, which means future readers can follow it without re-learning.

Defer the hard problems. Chroma local instead of pgvector. LangFuse Cloud instead of self-host. Ragas with a small golden set instead of a comprehensive eval framework. Each is the right choice for v1 and the wrong choice for prod. The README’s “missing for prod” section is honest about that.

What I noticed building my RAG app

  • The constraint to “match the team’s stack” did almost all the architectural work. Without it, decisions would have taken days; with it, each one was 5 minutes.
  • uv made setup trivial. No virtualenv activation, no pip, no requirements.txt. uv init + uv add + uv sync is the whole story.
  • The [build-system] + [tool.hatch.build.targets.wheel] packages = ["src/rag_app"] block in pyproject.toml is non-obvious. uv init doesn’t add it; you need it once you want a src/ layout and importable package.
  • All the “v2” deferrals (semantic cache, self-host LangFuse, pgvector) are documented in the README. Writing them down at decision time makes the trade-offs honest later.