Your AI agent just gave you a confidently wrong answer. The first instinct usually is blaming the model and looking for a better one. But before you do that, we want you to look closer into a more fundamental problem.
Did the agent have all the right context to get the answer correct?
In this article, we explore what context really means for AI agents, why more information can sometimes make them worse, and how better context engineering can turn an agent from usually right into reliably right (and we are not taking the confidence away).
When an agent gives a bad answer, what is our first instinct?
Maybe we need a more capable model.
Maybe we need a better prompt.
Maybe we need a larger context window.
But what if the model was never the problem?
“When people say “the AI is dumb,” what’s actually happened 8 times out of 10 is: the agent was never given the piece of information it needed to make the right call, and it did something reasonable but wrong with what it had.”
That is how one of our AI engineers put it. In her experience, what looks like an intelligence problem is often an information problem.
Let’s consider a simple example.
An AI support agent is asked to determine the eligibility of a refund for a customer. The model may be perfectly capable of applying the refund policy. But if it only receives the support ticket and nothing else, it does not know the customer’s order history, previous refunds, or account status.
It can guess. It can even produce a very convincing explanation for its guess. But give that same agent the relevant account record and refund history, and the solution changes completely.
The capability was never the bottleneck. It was the availability of information.
And this is where context engineering comes in.
What is “context,” really?
When we talk about giving an AI agent “context,” it is tempting to think about a prompt, a document, or a few retrieved chunks of information. But context is much broader than that.
Context is not simply what the agent knows. It is what the agent has access to right now.
Anthropic describes context as the collection of information made available to an LLM at the moment it generates its next output. For an agent, that can include system instructions, conversation history, tool definitions and outputs, retrieved information, external data, and other state available to the model.
That makes context a finite engineering resource.
An agent does not need to know every information your organization owns. It needs the information that can help it make the decision in front of it. This distinction is becoming more and more important as agents are becoming more capable.
Why do we need context engineering?
AI models can process enormous amounts of information, and context windows continue to grow. But bigger does not mean unlimited.
Like humans, LLMs can lose focus or become less effective when presented with too much information. More context can produce diminishing returns, and in some cases, too much irrelevant or conflicting information can actively make the result worse. (Anthropic 2025)
This creates a different engineering problem.
The question is no longer simply: “How do we give the model more information?”
It is: “What information should the model have for this particular task, at this particular moment?”
That is the shift from prompt engineering to context engineering.
Prompt engineering vs. context engineering
Prompt engineering asks: How should we tell the model what to do?
Context engineering asks: What should the model have access to when it decides what to do?
A well-written prompt cannot compensate for missing information.
As our engineer put it:
“A great prompt can’t compensate for missing information, but a good prompt on top of good retrieval is what turns ‘usually right’ into ‘reliably right.’”
If models have huge context windows, why not just give them everything?
If context is valuable, the obvious solution seems simple: Give the agent everything. After all, modern models have enormous context windows.
But this is where things get more complicated. Context has a cost, and more importantly, more context does not automatically mean better context.
As context grows, useful information can become buried under irrelevant information. Long-running agents can accumulate tool outputs, conversation history, intermediate results, and outdated information. Usefulness can decline as context becomes bloated, a phenomenon known as context rot. (Claude 2026)
One way to deal with this is summarization or compaction.
Maximize the signal-to-noise ratio
Our engineer identified two ways excessive context can hurt an agent.
The first is signal dilution. Bury one critical constraint inside ten pages of loosely related documentation, and the information may technically be present without being practically useful.
The second is false context. Stale or slightly incorrect information can be worse than having no information at all because the agent may trust it.
An agent without calendar access has to ask when a meeting is. An agent with access to an outdated calendar might confidently give you the wrong time.
So dumping an entire knowledge base into an agent’s context is not retrieval. It’s hoping. The goal is not to maximize the amount of information available to the model. It is to maximize the amount of useful signal available when the model needs it.
But do keep in mind that just ‘summarization’ is not the perfect answer.
Some recent production experiments suggest that retaining full history can sometimes outperform summarization, particularly when prompt caching changes the cost and latency equation.
So the lesson is not: Always summarize.
Nor is it: Always keep everything.
The better principle is: The goal isn’t less context. It’s better context.
What information does an agent actually need to do its job well?
Task definition: What does “done” actually look like? This includes edge cases, exceptions, and what the agent should not do. Most prompts define the happy path and leave the exceptions unclear.
State and history: What has already happened in this session or workflow? Without this context, an agent can repeat steps, lose track of decisions, or contradict what it did earlier.
Domain knowledge: The information a human in that role would simply “know,” such as policies, schemas, naming conventions, and organization-specific terminology. This is often the most overlooked piece of context.
Tool configurations: Simply telling an agent that a tool exists is not enough. It needs to know when to use it, what the tool does, and what can go wrong. Without knowing the capabilities of a tool, agents can easily misuse it.
Where does RAG help?
Retrieval-augmented generation changed how we give models access to external knowledge. Instead of forcing everything into a prompt, an agent can retrieve relevant information when it needs it.
But retrieval alone does not solve the context problem.
A September 2026 DataHub piece argues that enterprise agents can retrieve perfectly relevant information that is nevertheless untrustworthy, outdated or drawn from the wrong system. In other words, context quality depends on the quality and governance of the underlying knowledge, not merely the retrieval mechanism.
An agent can retrieve information that is relevant to the question and still make the wrong decision because that information is:
- outdated
- incomplete
- from the wrong system
- inconsistent with another source
- outside the agent’s permissions
- missing the surrounding business context
An agent can be perfectly grounded in the wrong information.
So the question is not simply: “Can the agent retrieve the information?”
It is: “Can the agent retrieve the right information, from the right source, at the right moment, and know how to use it?”
Just-in-time information retrieval
This is where context becomes dynamic. One of the most common mistakes developers make is treating context as a one-time setup step. They write one large system prompt containing everything the agent might ever need and call it done.
But an agent debugging a deployment needs a different context from that same agent triaging a customer ticket.
The system may be the same. The agent may be the same. The task is different. So the context should change with it.
A useful rule our engineering team practices is:
Remember what is stable. Retrieve what changes. Ignore what does not affect the decision.
If information is large, dynamic, or only relevant to certain tasks, retrieve it only when needed instead of keeping it in context permanently.
The rise of tools and protocols such as MCP makes this increasingly practical. Instead of loading an agent with everything upfront, relevant information and capabilities can be made available at runtime.
Context engineering best practices
Organization is key
Once we stop thinking of context as “a big pile of information” and start treating it as an engineered environment, organization becomes critical.
Not all information belongs in the same place. Not all information needs to persist. And not every task needs access to everything.
Separate durable information from temporary information
Some information should persist across tasks:
- business rules
- system architecture
- stable user preferences
- organizational policies
- core constraints
Other information is temporary:
- tool outputs
- intermediate results
- task-specific documents
- irrelevant conversation history
Compress deliberately
Long-running agents eventually accumulate context. That makes compression useful, but compression should not become an automatic reflex. Summarization can reduce context size, but it can also remove information that later turns out to matter.
Instead of asking, “How can we make the context smaller?” Try asking, “What can we safely remove without changing the agent’s ability to make the right decision?”
That distinction changes everything.
Isolate sub-contexts for specialized tasks
An agent working on one problem does not necessarily need to carry the entire history of another. Specialized tasks can benefit from specialized context.
A coding agent investigating a production error may need:
- the relevant code
- deployment history
- logs
- system architecture
- recent changes
It probably does not need an unrelated customer conversation from three hours earlier.
Keeping unrelated information out of the active context reduces noise and makes the agent’s working state easier to reason about.
Treat permissions as part of context
There is another layer that becomes increasingly important as agents gain access to enterprise systems.
It is: What information is this agent actually allowed to access?
Permissions should be treated as part of context architecture.
An agent’s memory and retrieval systems cannot simply be neutral information stores. They need to respect the same access rules that govern the underlying organization.
The right context is not everything that exists. It is everything the agent needs and is allowed to use for the task at hand.
Quick insights
- Context engineering isn’t finished when the agent gets the right answer. It’s finished when you can understand why it got the right answer, and diagnose why it got the wrong one.
- Give the model the smallest useful context. As discussed earlier, Anthropic explicitly recommends maximizing the signal-to-noise ratio of the context.
- Retrieve information with Just In Time retrieval methods. Here’s one of our favorite pieces on using MCP to do so.
- Treat context as dynamic, not as a one-time setup step.
- ✦ Remember (keep in the persistent prompt/state): things that are true across almost every task the agent does, identity, core constraints, stable preferences.
- ✦ Retrieve (fetch just in time): anything that’s large, that changes, or that’s only relevant to some tasks.
- ✦ Ignore: anything the agent can’t act on differently based on knowing it. If a piece of information doesn’t change the output, it’s just noise burning context budget.
Context engineering is an architecture decision
The more capable AI agents become, the less useful it becomes to think of them as isolated models. An agent is a system. Its reliability depends on the model, but also on:
- what information it can access
- how that information is retrieved
- which tools it can use
- what those tools return
- what it remembers
- what it forgets
- what constraints it
- operates under
- what it is allowed to access
- how all of that information changes from one task to another
This is why context engineering is becoming an architecture decision rather than simply a prompting technique. And it changes how we should think about improving agents. (Sourcegraph 2026)
Our engineer’s recommendation is straightforward:
“Retrieval and tools, in that order, before touching the model.”
A better model can give you a better guess when information is missing. Better retrieval means the model does not have to guess in the first place.
The real bottleneck may be the information supply chain
There is a broader lesson here for businesses building AI agents.
Reliability is not just a model problem. It is an information supply chain problem.
Leaders often approach AI with the assumption that “we need a better model,” much like saying, “we need better people.” But even the best skilled employee will struggle without access to the right systems, the right information, and clear goals.
The same goes for AI agents. Capability matters, but capability without the right information and constraints is unreliable.
The teams getting true reliability out of agents are not necessarily the ones with the newest model. They are the ones asking a more fundamental question:
What does the agent actually need to know to carry out this task, and how does it reliably retrieve that information when needed?
Better models matter. But better models do not eliminate the context problem. They make the context problem more important.
