notes on meowy

It all happens in the context window

Sat, 30 May 2026 00:00:00 +0000

Everything starts with one small fact, the model only ever sees what we put in front of it. It has no memory of my files, my last conversation, anything. The only thing it knows in the moment is whatever text we hand it in that one call. That text is the context window, and it is not infinite. There is a hard limit to how much fits.

And one thing worth noting is that the model doesn’t read words or characters, it reads tokens. A token is just a chunk of text, roughly three-quarters of a word on average, so “cat” is one token and something like “tokenization” is a few. This matters because the window limit is counted in tokens, not words, and so is the bill for the model. When we hear a model has a “128k context,” they mean 128,000 tokens. So tokens are the thing we’re actually budgeting when we decide what goes into that window.

The LLM is the smallest part of the AI app

Sat, 30 May 2026 00:00:00 +0000

Everyone obsesses over the model. Which one is smartest, which one just dropped, GPT vs Claude vs whatever came out this week. But the funny thing is, once I actually sat down and built something with an LLM, calling the model turned out to be one line of code, basically. I send it some text, it sends text back. That’s the easy bit.

The actual work, the stuff that we end up fighting day to day, is everything wrapped around that one call. What goes into the context window. What history to keep and what to throw away. What all of it costs in latency and money. And what to check before it goes in and after it comes back. Because the model will happily do something dumb if we let it.

The window isn't free

Sat, 30 May 2026 00:00:00 +0000

Since everything in the window is counted in tokens and I pay for those tokens, a big context window isn’t free space, it’s a budget. And it bills me in two currencies: money, and latency. (Latency is just the technical word for the wait, the time from hitting send to getting an answer back.) It’s tempting to treat the window like free real estate and paste everything in, the model has a huge one anyway. But every token in there costs me twice, once on the bill, once on the clock.

Trust nothing on either end

Sat, 30 May 2026 00:00:00 +0000

The last bit of work wrapping the model is what we check on the way in and on the way out. And the way I think about it is something I already do with my web3 programs, trust boundaries. A trust boundary is just the line where data crosses from somewhere I don’t control into somewhere I do, and the rule at that line is always the same, validate it, don’t assume it’s safe.

What history do we even keep

Sat, 30 May 2026 00:00:00 +0000

We need to understand that the model remembers nothing, it’s stateless (every call starts from zero, nothing carries over from the last one). Every single call is a blank slate, it has no idea what I asked it thirty seconds ago. So when a chatbot seems to “remember” the conversation, that isn’t the model remembering anything, it’s the app quietly re-sending the whole past conversation back into the window on every call. The “memory” is really just the app replaying history.

Embeddings

Thu, 28 May 2026 00:00:00 +0000

What an embedding is

For text-embedding-3-small, an embedding is a list of 1536 floating-point numbers between roughly -1 and 1. Same text → same vector, always. Different texts → different vectors. Texts with similar meaning → similar vectors.

"what is ownership in rust"      → [0.012, -0.034, 0.567, ..., -0.089]
"explain rust ownership"         → [0.011, -0.031, 0.572, ..., -0.085]   (very similar)
"how do I cook pasta"            → [-0.42, 0.81, -0.15, ..., 0.33]       (very different)

The embedding model was trained on billions of text pairs so that semantically similar text produces vectors close to each other in this 1536-dim space. You don’t see or interact with the training; you just call the model and get back numbers.

Evals for RAG

Thu, 28 May 2026 00:00:00 +0000

Why eval matters for LLM apps

Regular web app tests are deterministic: input X → assert output Y. RAG is non-deterministic: same question → potentially different answers, both “correct” in different phrasings. Asserting exact strings is useless.

You need metrics that measure quality in aggregate, even when individual responses vary. Without them, every prompt tweak is a guess. With them: “I changed X, faithfulness dropped from 0.95 to 0.78, revert.”

Ingestion

Thu, 28 May 2026 00:00:00 +0000

What the word means

“Ingestion” is borrowed from data engineering. Generically it means “pulling data into your system from somewhere else.” In RAG specifically, it means:

Source documents → chunks → embeddings → stored in a vector database.

Once ingestion is done, your system can answer questions about those documents. Without ingestion, the system has nothing to retrieve from.

What ingestion does in my RAG app

In my RAG app, src/rag_app/ingest.py is the entire ingestion script. It runs once (or whenever my notes change). What it does, in order:

Stack decisions for a RAG app

Thu, 28 May 2026 00:00:00 +0000

The starting point

The goal was a RAG service over a folder of markdown notes, with one binding constraint: the stack should mirror a real team’s production stack closely. That constraint did most of the architectural work. Without it, you have infinite freedom. With it, every choice gets one question: “does the team use it?”

The stack decisions, in order

These got locked in roughly in this sequence:

1. Python 3.13 + uv (language + toolchain)

Team uses Python 3.13. uv is the modern toolchain - one binary that replaces pip + virtualenv + pyenv + poetry. Picked it because it’s the only Python tooling that feels like cargo. One command for everything.