What history do we even keep

We need to understand that the model remembers nothing, it’s stateless (every call starts from zero, nothing carries over from the last one). Every single call is a blank slate, it has no idea what I asked it thirty seconds ago. So when a chatbot seems to “remember” the conversation, that isn’t the model remembering anything, it’s the app quietly re-sending the whole past conversation back into the window on every call. The “memory” is really just the app replaying history.

The easiest way to see it is to watch what actually gets sent each turn:

turn 1, I send:
  user: what's the capital of France?
turn 1, model replies:
  Paris.

turn 2, the app rebuilds and sends:
  user: what's the capital of France?
  assistant: Paris.
  user: what's its population?      ← only this line is new

By turn 2 I’m re-sending turn 1. By turn 10 I’m re-sending turns 1 through 9. The model answers “its population” right not because it remembers Paris, but because Paris is sitting in the text I just handed it. Delete that line and it has no clue what “its” even refers to. So the “history” is just those role-tagged messages from earlier, rebuilt and replayed every call. And two things fall straight out of this, the bill grows with the conversation since every turn re-sends all the turns before it, and the transcript keeps growing until it bumps into the window limit. Which is exactly why I can’t just replay forever, and why this needs actual handling.

So the real question is, where does the conversation actually live? Not in the model. It lives in storage I control, and I replay it back in every single time.

Now this memory part actually is 2 things:

First, the session. That’s one conversation, the back-and-forth of a single chat, plus whatever scratch state rides along with it. It’s short-lived, usually on a TTL or time to live (an expiry timer, so when the chat goes quiet the session just clears itself out). It lives somewhere fast, like redis or a row in a database keyed by some session_id. My little RAG app does exactly this, a SQLite table holding the chat history.

Second, memory. That’s the stuff I want to keep across conversations, not just this one chat. Facts worth holding onto, like who the user is or what they told me last week. It’s long-lived, it survives after the session is over and it’s usually put down to the bits that matter instead of the whole transcript. It lives in a persistent database or a vector store keyed by the user, and pulling the relevant parts back out.

Side by side it’s easier to keep straight:

So why does everyone blur the two?

Mostly because both end up as text in the prompt, so on the surface they look identical. And also because the tooling doesn’t help, langchain literally calls its conversation-buffer stuff “Memory” when most of it is really just session handling. The line that actually separates them is scope and lifetime, so session is one conversation and short-lived and memory is across conversations and sticks around.

And the reason this is real engineering and not just two definitions is that I can’t keep dumping the whole history into the window forever. It grows every turn, and sooner or later it blows past the limit and costs more on every call. So both sides need a way to stay small. For a session, that usually means a windowed buffer (just keep the last few turns) or a summary buffer (summarize the older turns into a short recap and keep the recent ones as they are). For memory, it means not storing every message at all, just the facts worth keeping and pulling back only the relevant ones when they matter (which, again, is RAG).

Honestly I’ve only really built the session side so far, the SQLite chat history in my RAG app. I haven’t built long-term memory yet. So I know this split more from it tripping me up than from shipping it. But the split itself is the part that matters.

part of the llm is the smallest part, my notes on the work around the model call.