meowy

notes on what i build and what i learn

← Field map

The window isn't free

· note · budding · about wiki-rag

Since everything in the window is counted in tokens and I pay for those tokens, a big context window isn’t free space, it’s a budget. And it bills me in two currencies: money, and latency. (Latency is just the technical word for the wait, the time from hitting send to getting an answer back.) It’s tempting to treat the window like free real estate and paste everything in, the model has a huge one anyway. But every token in there costs me twice, once on the bill, once on the clock.

The money part is simple. I pay per token on every single call. So if I’m resending the whole conversation history each turn, the bill grows with the conversation and not just with the question. A chatty app that replays everything can quietly get expensive.

The latency part is where it gets interesting, and it helps to know how the model actually spends its time. Underneath, the model only ever does one thing: it looks at all the tokens so far and predicts the next one, then adds that token to the pile and does it again. The whole answer is just that one move, repeated. And it runs that move in two phases:

  • prefill (the reading phase): first the model chews through everything I sent it, the system prompt, the context, the history, the question, to work out what the very first word of its answer should be. It already has all that text in hand, so it can process the whole thing in one pass. The more I send, the longer this takes, and this is the wait before the first word ever shows up.
  • decode (the writing phase): from there it goes one token at a time. It takes everything so far (my input plus whatever it’s written already), predicts the next word, sticks it on the end, then repeats with that new word now included, again and again until it decides to stop. Each step needs the previous word to already exist, so it can’t rush ahead, it’s strictly one after another. A longer answer just means more of these steps.

Those two phases line up with the two numbers people actually measure:

  • TTFT (time to first token): how long until the first word shows up. That’s basically the prefill time, so it’s driven by how big my input is.
  • TPOT (time per output token): how fast the words come after that, the speed of the stream. That’s the decode speed.

So the whole wait is roughly TTFT + (number of output tokens × TPOT), the time to start plus the time to finish typing. A giant context blows up TTFT (the wait before anything shows up at all), and a long answer blows up the back half.

Before the cache makes any sense, there’s one thing about how the model picks the next token that I glossed over. I keep saying it “looks at all the tokens so far,” and the way it does that looking has a name, attention. For the token it’s about to write, the model basically asks every earlier token “how relevant are you to what I need right now?” and leans on the ones that matter most. To pull that off, each token gets turned into three little things:

  • a query: what the current token is looking for
  • a key: what each earlier token is about, so it can be matched against that query
  • a value: the actual content each earlier token carries

So the model takes the current token’s query, checks it against every earlier token’s key to see what’s relevant, and mixes in the values of whatever matched. That’s the whole move, and it’s worth holding onto because the key and value are exactly what the cache is named after.

Now the trick that makes decode bearable at all, the KV cache. Here’s the thing, a token’s key and value never change once they’re computed. Token 3’s key and value are the same whether the model is writing token 10 or token 500. So recomputing them on every single step is pure waste. The KV cache is just the obvious fix, compute each token’s key and value once, save them, and reuse them after that. Without it, writing token number 500 would mean re-chewing the first 499 from scratch every time. (It’s the K and V that get cached and not the Q, because I only ever need the current token’s query, there’s nothing to reuse there. Nice thing to know if anyone asks.)

The bit that’s easy to miss, and the bit that matters, is that this saved work covers the input too, not just the answer so far. When the model first reads my prompt (the prefill phase up above), it computes a key and value for every token in it, the system prompt, the context, the history, the question, and all of that lands in the cache. So the cache is really the model’s processed notes on everything it has read and written, not just the reply it’s busy typing.

And that’s exactly what powers prompt caching. Normally that prefill work gets thrown away when the call ends. Prompt caching is the provider hanging onto it instead, so when the start of my next prompt is byte-for-byte identical (like a big fixed system prompt), it loads the already-processed version instead of charging me to chew through it again. That cuts both the bill and the TTFT, and it’s one of the biggest wins there is.

The catch is it only works as a prefix, matched from the very first token. Change one token early and nothing after it can be reused. Which is the real reason the stable stuff (the system prompt, fixed context) goes at the front and the variable stuff (the actual question) goes at the back, same placement point from earlier, now with money attached to it.

One more pair worth knowing, because they pull against each other: latency (how long my one request takes) versus throughput (how many tokens per second the system pushes out across everyone using it at once). Servers batch lots of requests together to get throughput up, but that batching can add a bit to any single request’s wait. Tune for one and other usually pay for it.

And averages lie. If 99 requests are quick and one hangs for ten seconds, the average still looks fine while that one user had a miserable time. So instead of averaging, people line every request up from fastest to slowest and look at where a given one lands. That’s a percentile. p50 is the one in the middle, the typical request, half are faster and half are slower. p99 is way out near the slow end, only 1 request in 100 is slower than it. That slow end is called the tail (so “tail latency” is just how bad it gets for the unlucky few), and it’s what users actually feel and complain about, since nobody notices the fast ones. The catch is that long contexts make the tail worse, not just the average, so a big prompt can look totally fine on average while the p99 quietly gets painful. Which is exactly why “fast on my machine” means nothing here, one good run is just a p50. The p99 is the real story.

It’s like I said in the context window note, past a point more context doesn’t even buy a better answer. So I’d be paying more, waiting longer and getting something worse. Not really a great trade.

So the real question is never “does it fit,” it’s “is it worth the tokens.”

There are a bunch of methods for that:

  • route by difficulty: send the easy stuff to a small, fast model and save the big slow one for when it’s actually needed (people call this cascading).
  • shrink the input: retrieve a few relevant chunks instead of pasting whole documents, trim or summarize old history. Less input means shorter prefill, faster TTFT, smaller bill.
  • shrink the output: cap the max tokens and ask for a concise answer, since decode (the actual writing) is a big chunk of the total wait.
  • prompt caching: reuse the fixed parts of the prompt so I’m not paying to re-process them every call.
  • streaming: push each token to the screen the moment it’s written (server-sent events under the hood). It doesn’t make the total any faster, but the user sees words at TTFT and it feels fast. Pure perception, and a big UX win.
  • reasoning models: the ones that “think” (quietly generate a pile of hidden tokens) before answering are smarter but burn more tokens and more time, so switching that on is its own fast-or-smart call.

The part that makes this click for me is that the context window is basically working memory, like RAM. I don’t load the entire database into memory to answer one query, I index it and pull only the rows I need. Same theory here as the window is small, so I only spend it on what actually matters.

The first time this got real for me was when I built my first RAG app. Before I tuned anything, I wired up LangFuse just to watch the token counts and the cost per call. Because I wanted to see what I was actually spending before guessing at fixes. And I capped how much could go into the prompt in the first place. The same don’t pay twice instinct showed up over in the indexing too, I cached the embeddings in Redis so re-running ingestion didn’t re-embed the same notes and bill me all over again. So treat the window like it costs something, because it actually does.

part of the llm is the smallest part, my notes on the work around the model call.