meowy

notes on what i build and what i learn

← Field map

The LLM is the smallest part of the AI app

· note · budding · about What RAG actually is, wiki-rag

Everyone obsesses over the model. Which one is smartest, which one just dropped, GPT vs Claude vs whatever came out this week. But the funny thing is, once I actually sat down and built something with an LLM, calling the model turned out to be one line of code, basically. I send it some text, it sends text back. That’s the easy bit.

The actual work, the stuff that we end up fighting day to day, is everything wrapped around that one call. What goes into the context window. What history to keep and what to throw away. What all of it costs in latency and money. And what to check before it goes in and after it comes back. Because the model will happily do something dumb if we let it.

I already wrote about RAG, and RAG is really just one slice of this, the part about getting the right knowledge into the window. These notes are the job around that.

Swapping the model is one line of code, and a better one seems to drop every other week. That part is basically a commodity now, I can move from one to another in an afternoon.

What doesn’t swap out is everything around it. Whether the right stuff actually made it into the window. Whether the conversation carries over from one call to the next. Whether the whole thing is fast and cheap enough to put in front of real people. And whether it can be trusted not to do something dumb on the way in or the way out. That’s the work that decides if the app is any good, and none of it is the model itself.

That’s what it means when we hear context engineering is most of the job now. The model is the easy, finished, swappable piece. The real work is the plumbing wrapped around it.

The four parts: