Working Memory and State in Agents
An agent that perceives, decides, and acts but remembers nothing is a goldfish on an adrenaline rush. It does a lot, and then it forgets it did any of it. Every loop restarts with a blank slate, which is charming for a while and useless for real work. Memory is what turns a series of clever moments into a sustained project.
This article unpacks the two very different things people mean when they say "memory" in an agent context: the context window that holds immediate data, and the long-term store that persists across sessions. Both shape what an agent can realistically accomplish.
Two Flavors of Memory
Confusion around memory usually comes from conflating these layers. Let's separate them cleanly.
- Working memory: the context window — everything the model can see right now as it decides the next action. It is fast, finite, and vanishes when the task ends.
- Long-term memory: persistent state — documents, databases, conversation summaries — stored outside the model and loaded back when relevant.
Think of it in human terms. Working memory is what you hold in your head while solving a problem in a single sitting. Long-term memory is your notebook, your files, and the summarized history you consult before resuming.
"Working memory is fast and fragile. Long-term memory is slow and durable. An agent needs both, played against each other."
Working Memory and the Context Window
The context window is the model's working memory, with a hard size limit measured in tokens. Every turn, the agent must pack its instruction, the history so far, the user's message, tool results, and any fresh observations into that finite budget. It is a constant, uncomfortable squeeze.
Short windows force aggressive sacrifice. The agent must drop older messages, truncate long tool outputs, or summarize what happened before continuing. Long windows give breathing room, but they are not a free lunch — two hidden costs follow:
- Cost: every token in the window is billed on every call. A sprawling window makes each iteration pricier.
- Latency: attention scales with length, so bigger windows mean slower generations and fuzzier recall of details buried in the middle.
In practice, waste is the enemy. Filling a huge window with ten repeated search results is not smart memory use; it is hoarding. Good working-memory management keeps the window lean and relevant.
Managing the Squeeze
Because the window is finite, engineers spend real effort deciding what earns a seat. Three techniques dominate.
- Truncation: cutting long tool outputs to their meaningful head, keeping summaries of what matters.
- Summarization: periodically condensing older conversation into a short rolling recap, trading detail for span.
- Selective retrieval: fetching only the specific facts or document chunks relevant to the current goal, instead of loading everything.
Done well, these keep effective memory shallow and sharp. A common failure is loading an entire chat and praying the model sorts it out; the model can read a lot, but it still loses threads in noise.
Long-Term Memory and State
Beyond a single session, agents need somewhere to park what they learn. Long-term memory is whatever persistent store you point them at, and it wears a few common hats.
- Conversation history: past sessions, stored and summarized so the agent can pick up where it left off.
- User and task profiles: preferences, constraints, and standing instructions that apply across interactions.
- Work product: files, notes, drafts, and decisions generated during earlier loops.
- Repositories and knowledge: vector stores of documents, manuals, and code the agent can search when answering.
Because the context window is too small for all of this, the agent treats long-term storage as a library: it queries the relevant slice at need, reads it into working memory, then acts. Retrieval quality, not storage size, determines how smart the agent feels.
State Is More Than a Notebook
"State" casts a wider net than memory and is easy to forget. It includes the agent's place in a task — what has been done, what remains, which decisions are made. Some of that lives in the context window, but durable workflow state should survive even a crash.
Consider a multi-step purchase workflow. Without state, a failure mid-way strands the user with half a booking and no idea why. With explicit state — a task record marked "awaiting payment," "steps 1 and 2 complete" — the agent can resume, explain, or roll back. Robust agents persist their progress, not just their chat.
Forgetting Is a Feature
Engineers rightly worry about remembering, but forgetting deserves design too. Unbounded memory is a liability: stale preferences override current ones, obsolete context crowds the window, and privacy obligations go unmet.
- Retention limits: how long data is kept before it is purged.
- Consent and erasure: honoring a user's right to delete their data, including what the agent learned.
- Versioning and override: making sure a newer preference outranks an older one instead of silently conflicting.
Good memory systems have hygiene policies, not just capacity. An agent that remembers too much can be as broken as one that remembers too little.
Making Memory Tactical
Teams turn these ideas into working systems along a familiar arc. They start with just the context window, letting the agent survive single sessions. Then they add a summary file per conversation, an early form of long-term memory. From there comes a vector index over documents, then structured state records for ongoing tasks.
Figure 1. Working memory is a finite sieve. Information enters, participates, and departs to make room.
The pragmatic rule is to store what you would actually want to retrieve later, and index it well. Storing everything and retrieving nothing is decoration. The value lives in the retrieval path, which is where most of the engineering effort belongs.
Memory Shapes Scope
Here is the honest punchline: memory determines ambition. An agent with only working memory can handle a bounded, single-sitting task — answer a question, transform a file, run one workflow. The moment a goal spans hours, requires grounding in a large corpus, or must survive interruptions, long-term memory becomes mandatory.
So before you promise an agent that it will "remember everything," set expectations. Ask what it needs to remember, for how long, and how it will find the right memory at the right moment. Those answers, not the model's raw capacity, will decide whether the agent delivers on its promise.
Figure 2. Long-term storage is a library the agent consults; only what is relevant enters the working window.
Bringing the Series Together
We have now assembled the full agent from the ground up. Perception feeds facts. Decisions choose actions. Tools give it hands. And memory — working and long-term — lets it learn, persist, and scale beyond a single clever trick.
The pieces interlock. A well-defined loop (article three) needs tools (article four) to act meaningfully. Tools generate output that must be stored in working memory, then distilled into long-term memory when it overflows. And the whole machine earns its "agent" label precisely because it can chase a goal with autonomy, unlike the chatbots of article two.
"An agent is only as ambitious as its memory. Extend its memory, and you extend what it can do."
Build the pieces in order, respect the boundaries, and you will have something that does not merely answer questions — it remembers, persists, and works toward a goal long after the first message is sent.
