It all happens in the context window

Everything starts with one small fact, the model only ever sees what we put in front of it. It has no memory of my files, my last conversation, anything. The only thing it knows in the moment is whatever text we hand it in that one call. That text is the context window, and it is not infinite. There is a hard limit to how much fits.

And one thing worth noting is that the model doesn’t read words or characters, it reads tokens. A token is just a chunk of text, roughly three-quarters of a word on average, so “cat” is one token and something like “tokenization” is a few. This matters because the window limit is counted in tokens, not words, and so is the bill for the model. When we hear a model has a “128k context,” they mean 128,000 tokens. So tokens are the thing we’re actually budgeting when we decide what goes into that window.

So the whole thing becomes deciding what goes into that limited space. And honestly bunch of things are fighting for it. The window might be holding:

the system prompt, the rules and the format I want back
whatever knowledge I pulled in (the RAG chunks)
the conversation so far (the session), and any longer-term memory
the tools I’ve handed it, plus whatever those tools handed back
a few examples of what I want, if I’m showing it the format (few-shot prompting)
and finally, the actual question

All of that shares one window, so every extra thing I put in pushes something else out. And it isn’t free either, the more I stuff in, the slower and pricier every single call gets. That part deserves its own note: the window isn’t free.

This is actually context engineering. It’s the mature version of prompt engineering. Prompt engineering was about wording one message nicely. Context engineering is about putting together the whole input, from a bunch of different sources, into that limited space.

And most of this isn’t clever writing, it’s plumbing. Prompt engineering felt like finding the magic phrasing. Context engineering is more like code: retrieve these chunks, grab the last few messages, drop in whatever the user told me before, trim it down to fit, then hand the whole thing over. It gets assembled fresh on every call, usually by code I wrote, not typed out by hand.

And it isn’t even one blob of text. The API takes the input as roles, a system message (the standing rules, the format I want back, the persona), the user’s messages, the model’s own past replies (the assistant messages), and on newer APIs a tool role too (whatever a tool handed back).

The thing that surprised me is that these roles aren’t actually a wall. Under the hood the whole list gets flattened into one single stream of tokens, with little marker tokens saying where each part starts and stops. It ends up looking roughly like this (this particular shape is OpenAI’s chat format):

<|im_start|>system
Answer only from the provided context.<|im_end|>
<|im_start|>user
What's the refund window?<|im_end|>
<|im_start|>assistant

That’s all a role really is, a chunk of text wrapped in marker tokens. There’s no real barrier between them, it’s one sequence the model reads start to finish. So the reason it treats the system message as the boss isn’t some hard rule baked into it, it’s just that it was trained that way, to lean on the system part when the system part and the user part disagree (people call this the instruction hierarchy, system outranks user, and user outranks tool output). Which means the line between my instructions and the user’s data is a learned habit, not a guarantee. And that’s the whole reason prompt injection is even possible, more on that in trust nothing on either end.

And that’s exactly why deciding what goes in which slot is part of the job. Instructions go in system so they carry the most weight. The history goes in as alternating user and assistant messages, and that replay is literally how a chatbot seems to “remember” anything. The actual question goes last, where the model pays the most attention. The tricky one is retrieved context, do I drop a RAG chunk into the high-trust system slot, or fence it off somewhere lower where it’s safer but trusted less? There’s no free answer. Which is exactly why it’s the real job and not a footnote.

The main work for managing that window mostly comes down to four moves. And really they’re all answers to the same question, this stuff won’t all fit, or it shouldn’t all be in there, so what do I do with it?

write: push it out of the window. Save the stuff to storage so the window isn’t carrying it. This shows up two ways, a scratchpad (the agent jots a running plan or some notes to a file or a db mid-task instead of holding it all in its head) and memory (keeping facts around across conversations). This is the long-term memory bit in what history do we even keep.

select: pull only the relevant part back in. This is retrieval, and it’s the same instinct whether it’s RAG chunks, the few past memories that actually matter, which tools to even hand it, or which examples to show. The keyword is relevance, only what earns its place. This is the move I’ve actually built.

compress: shrink what I do keep. Same information, fewer tokens. Summarize the older turns into a short recap, boil down a giant tool output, trim the fat. Built this one too.

isolate: split it across separate windows. Instead of one giant context doing everything, an orchestrator agent hands sub-tasks off to smaller sub-agents, and each one gets its own fresh, focused window. A research sub-agent only sees research stuff, a coding sub-agent only sees code, so they never muddy each other. And the orchestrator only gets back the result (a little summary), not the sub-agent’s whole working context, so the main window stays clean.

What it buys is dodging the exact failure modes I mentioned, unrelated junk can’t cause context distraction or clash if it’s sitting in a totally different window. Each window stays small, so every call is faster, cheaper and more focused, and the sub-agents can run in parallel.

The hard part, and the reason it’s real engineering and not just a diagram, is the hand-off. Sub-agents can’t see each other’s windows, so if a task actually needs shared understanding, splitting it just makes them duplicate work or make decisions that clash. Plus the coordination overhead, more calls in total, and then stitching the results back together so they read as one answer. The ones to look at here are Anthropic’s multi-agent research setup (a lead agent spawning sub-agents) and Claude Code doing the same through a Task tool.

And these aren’t a menu where I pick one, a real app does all four at once, write state out, select what’s relevant back in, compress what stays, and isolate across agents when one window just can’t hold the whole job.

Honestly though, I’ve only really done select and compress so far, pulling in the right chunks and trimming what I don’t need. write and isolate I know more as concepts than from building them, isolate especially, since that’s agent territory I haven’t gotten into yet. But I will soon, on my second project. They’re all the same instinct anyway, give the window only what earns its place.

And then two smaller things that matter more than they sound. Where I put stuff: models pay the most attention to the start and the end of the window and kind of zone out in the middle (the famous lost-in-the-middle problem), so the important bits shouldn’t be buried. And formatting it clearly, so the model can tell my instructions apart from the data.

And the part that’s easy to miss is that a full window can be worse than a half-empty one. More context doesn’t automatically mean a better answer. There are actually named ways it goes sideways. Pack in things that are only sort of related and the model gets pulled off course, that’s context distraction. Put two chunks in there that quietly contradict each other and it has to pick one, maybe the wrong one, that’s context clash. Let something false slip in early, like a bad retrieved chunk, and it sits there messing up everything after it, that’s context poisoning. So a lot of context engineering isn’t about fitting more in, it’s about keeping slop out.

And that’s the whole point really as the model is fixed, I can’t change how it thinks. The only real leverage I’ve got is what goes into that window. So that’s where the work actually is.

part of the llm is the smallest part, my notes on the work around the model call.