Skip to content
VAR/LOG/INFO
Context Windows: The Limit AI Models Can't Simply Outgrow
AI

Context Windows: The Limit AI Models Can't Simply Outgrow

Million-token context windows don't end the problem: long inputs cost more, run slower and get read less reliably. How to spend the token budget well.

Model announcements love one number: the size of the context window. A few years ago it was a few thousand tokens, today the big models advertise hundreds of thousands or even a million. It’s tempting to read that as “just give it everything”, and that’s exactly the mistake that makes AI tools slower, more expensive and less accurate than they need to be.

Tokens, briefly

A context window is the maximum amount of text a model can work with in one request, measured in tokens. A token is a chunk of text, often a piece of a word. For English a common rule of thumb is about four characters per token, or roughly 750 words per 1,000 tokens. Other languages, code and unusual formatting usually need more tokens for the same amount of content.

The important part is what counts. It isn’t just your question. The system prompt, the whole conversation history, every attached file, every result returned by a tool and the answer the model is writing all share the same window. In an agent that reads files and runs commands, tool output alone can fill most of it.

When the window is full, something has to go. Chat apps typically drop or compress the oldest messages, which is why a long conversation sometimes “forgets” a decision made at the beginning.

Storage or working memory?

The most useful mental model is to stop treating the context window as storage and start treating it as working memory. Storage is where you keep everything just in case. Working memory is the small set of things you need for the task in front of you, and the more you cram into it, the worse you get at using any of it. Seen that way, the size of the window is less interesting than what you put into it.

Three reasons a bigger window doesn’t solve it

Reliability drops with length. In 2023 the paper Lost in the Middle showed that models find information near the start and end of a long input much more reliably than information buried in the middle. In 2025 Chroma tested 18 models, including GPT-4.1, Claude 4 and Gemini 2.5, and came to the same conclusion: performance becomes less reliable as input grows, even on simple tasks. They called it context rot. A model that accepts a million tokens doesn’t treat all of them equally.

Cost scales with every request. API pricing is per token, and the whole context is sent again with every message. A 200,000-token codebase pasted into a chat isn’t paid for once, it’s paid for on every follow-up question. Prompt caching softens this a lot, for example Anthropic charges cached input at a tenth of the normal price, but it only helps when the beginning of the prompt stays identical.

Latency grows too. The model has to process the whole input before it writes the first word of the answer. For interactive tools the difference between a short and a huge context is felt on every request.

Spending the budget well

If the window is a budget, the goal is to spend it on what actually changes the answer. A few practices that work regardless of which model or provider you use:

  • Send the relevant part, not the whole thing. The function and its tests rather than the whole repository, the relevant chapter rather than the whole manual.
  • Retrieve instead of preloading. With RAG, or tools connected through MCP, the model fetches a document only when it needs it, instead of carrying everything from the first message.
  • Put the important things at the edges. Instructions and key facts belong at the start or at the end of a long prompt, not in the middle of a pasted document.
  • Keep the stable part stable. If you use the API, put the unchanging system prompt and reference material first, so prompt caching can reuse them.
  • Restart long sessions on purpose. Ask the model to summarise where things stand, then continue in a fresh conversation with that summary. Most coding agents now do some form of this automatically, often called compaction.

Choosing what to send

Windows will keep growing, and models will get better at using long inputs evenly. But the economics stay the same: every token sent costs money and time, and every irrelevant token is a chance for the model to focus on the wrong thing. Choosing what to send will remain one of the most practical AI skills there is.