A colleague who forgot every prior conversation the moment it ended would not last long in most jobs. Yet a lot of business AI works exactly that way. The session closes and the context disappears with it. A consumer firing off one-off questions may barely notice, but a company running a process that lasts longer than a single sitting hits the limit almost immediately.
Real work carries forward from one session to the next, and a decision taken on Thursday usually rests on one made on Monday, with the thinking behind it needing to survive in between. An AI that wipes itself clean when a session ends can perform well inside that session and still contribute nothing to a task running over a week.
Enterprise AI has quietly changed what it demands from a model. For years, models were judged largely on how much knowledge they had absorbed. Businesses now need them to keep the right information in view as work unfolds, a capability that calls for a different kind of efficient engineering. To date, the ability to hold this level of persistent knowledge has not been achieved due to the large amount of memory (GPU’s), compute, and cost required to scale such a capability. Maintaining an acceptable latency and level of model hallucination becomes a challenge with concurrent usage when large context is being utilised. There is increasing demand for such knowledge capabilities and not enough memory and compute to go around. So, this begs the question: how do you achieve more accurate intelligence at scale with less infrastructure and footprint?
The Desk and the Filing Cabinet
Two things get bundled under the heading of memory, and they behave differently enough that the rest of this depends on keeping them apart.
Start with the context window, which is how much a model can take in and reason over in a single pass. It works like the surface of a desk. A small one takes a few pages at a time, so everything else waits in a drawer and gets fetched and cleared as you go, while a large one lets the whole case file lie open at once as you work across it. What the desk holds gets swept off at the end of the day, every day.
Persistent memory is the filing cabinet beside the desk. The system chooses what to store and pulls it back out when it becomes relevant, so something that happened this week can inform what it does next week. That store carries across sessions, and it needs its own decisions about what to save, how to organise it and when to bring it back.
Much of the visible engineering race has gone into the desk. Windows that once held a few thousand tokens now run to hundreds of thousands, and the largest models from OpenAI and Anthropic top out around a million.
The term for this work, ‘context engineering’, came from Shopify’s Tobi Lutke and was popularised by Andrej Karpathy in mid-2025. The core skill in any serious AI application, Karpathy wrote, is filling the window with the right information for the next step. His analogy was hardware – the model as processor, the window as its working memory. Practitioners adopted the term fast, because they had been circling the idea without a label for it.
Having a Bigger Desk Does Not Solve the Problem of Memory
Here is where the obvious move – just keep enlarging the desk – runs into trouble.
Stanford researchers showed in 2024 that when the information a model needs sits in the middle of a long input, its accuracy drops sharply against the same fact placed at the beginning or end. They called it ‘lost in the middle’, and it held even for models purpose-built for long context. Putting more in front of a model, it turns out, is not the same as the model making reliable use of it.
The effect has held up as other teams have gone looking for it, and a 2025 study running 18 frontier models against progressively longer inputs found performance degrading well before the window was even full, a pattern its authors named context rot. Enlarging the desk and getting a model to reason cleanly across everything on it are two separate problems, and the first does nothing to solve the second.
Curation Earns More than Capacity
Context engineering has grown into a discipline on the back of this, with a 2025 survey drawing on more than 1,400 papers setting out its methods and Gartner advising clients in July 2025 to prioritise context over prompts. The craft moved on from wording a sharper instruction to assembling a sharper set of inputs for the model to work from.
A larger desk raises the stakes on what you choose to put on it. Tip an entire document store onto the surface and the few pages that matter end up lost among noise, contradictions and superseded versions, where a tighter selection of what actually bears on the task lets the same model perform markedly better. The judgement lies in what belongs in front of the model, how it is framed, when it appears, and what stays in the cabinet.
This is what RAG (retrieval-Augmented Generation) was built for: chopping documents into fragments and retrieving the ones that look relevant, to work around a window too small to hold the whole thing. Say a contract dispute turns on one clause on page 200. The usual approach chops the contract into fragments and lets a retrieval system fetch whichever ones look relevant. Score page 200 as irrelevant and the model never sees the clause.
A window sized for the whole contract skips the fetching step entirely and gives the model the clause along with everything around it. Whether the model then reads a long input well is the reliability problem from the previous section, and it is a better problem to be left with than a retrieval system quietly deciding the clause was not worth passing along.
Inside a Session, and After It
Long context is what lets an AI agent work across a long job rather than a single exchange. An agent handling a multi-day compliance review or a supplier onboarding keeps the whole task in the window as it runs, so each step draws on the full state of the job. A long enough window can substitute for memory within a session.
That is also where the resemblance stops. Whatever the system should recall next week, once this session has closed, has to live somewhere the window is not.
Building that kind of memory brings a different set of problems, many of which the industry has only begun to confront. My training in psychology and neuroscience makes the comparison with human memory difficult to miss. We do not retrieve a perfect recording of an event. Each time we remember it, we rebuild it, and small errors can gradually become part of the accepted version. A machine memory can develop a similar problem when stored information is repeatedly summarised, merged or rewritten. Over time, the record can move further from the original event unless someone checks it, corrects mistakes, and removes information that has become unreliable.
Many companies deploying these systems have not hit these problems yet, and few are close to solved. What I would watch in a vendor is not the size of the window they advertise. A model that holds 10 million tokens is worth little if most of what it keeps is outdated, irrelevant, or wrong. Its answers can look well-sourced while resting on material the business should never have trusted. What earns the money is the judgment about what to keep and the ability to accurately reason over all the information, on a fraction of the infrastructure cost and footprint. That is doing more with less, and not all bigger windows come with this true capability.
