How LLM Context Windows Actually Work (and Why Bigger Isn't Always Better)

Talk to a large language model long enough and you hit a wall: it loses the thread from the start of the conversation, or it rejects a document for being "too long." That wall is the context window — the amount of text the model can hold in view while it works. It's one of the most quoted numbers on a model card and one of the most misunderstood. Here is what a context window actually is, the machinery underneath it, and why a bigger number on the spec sheet doesn't automatically buy you a better model.

What a context window actually is#

Start with the unit. LLMs don't read characters or words — they read tokens. A token is a chunk of text, roughly three to four characters of English, so it's often a short word or a piece of a longer one. "Tokenization" is one token; "the" is one; a rare or non-English word may split into several. As a rough rule, 1,000 tokens is about 750 words of English — and code, punctuation, and non-English text tokenize less efficiently, so the same idea can cost more tokens.

The context window is the maximum number of tokens the model can process in a single pass. The critical detail most explanations skip: it's shared. Your system prompt, the conversation history, any documents or tool outputs you paste in, and the model's own reply all draw from the same budget. Fill it with a long input and you leave less room for a long output.

One distinction clears up most of the confusion. The context window is not the model's memory of what it learned. What it learned is baked into its weights, frozen at training time. The context window is working memory — a whiteboard wiped clean at the start of every request. A model doesn't "remember" your last chat unless that text is fed back into the window. Weights are long-term memory; context is short-term.

By mid-2026, very large context windows have become a headline feature across the frontier. Several flagship models — proprietary and open-weight alike — now advertise windows measured in the hundreds of thousands of tokens, and some of the largest reach around a million. A million tokens is roughly 750,000 words — a stack of long novels, or a mid-sized codebase, in a single prompt. But the field hasn't settled on a single number: some flagship models top out well below a million (a few hundred thousand tokens is common), and smaller, cheaper models cap lower still — because, as we'll see, capacity is anything but free.

How the window actually works: attention and the KV cache#

The reason a context window has a hard limit — and the reason a bigger one costs so much — comes down to the core operation inside a transformer: self-attention.

When the model processes your text, every token "attends to" every other token: it weighs itself against all the others to decide which ones matter for predicting what comes next. That all-pairs comparison is exactly what lets the model connect a pronoun on page 40 to the name introduced on page 1. It's also expensive. Double the number of tokens and you don't double the work — you roughly quadruple it, because the number of pairwise comparisons grows with the square of the length (n tokens produce on the order of n² pairs). This quadratic scaling is the tax at the heart of every long-context model.

The second cost is memory. To avoid recomputing everything each time it generates a new token, the model caches the intermediate attention values for every token it has already seen — the KV cache (key–value cache). That cache grows linearly with context length and has to live in fast accelerator memory. At long context, the KV cache — not the model's weights — is often what fills up the GPU, and each generation step has to read the whole cache back, so long outputs become limited by memory bandwidth rather than raw compute. This is why serving a one-million-token request is dramatically more expensive than a ten-thousand-token one, and why providers charge a premium for long inputs.

There's also a subtlety in how models got these long windows. A model trained at a given length can't simply be handed ten times more text — it has no learned sense of positions it never saw during training. Extending context is real engineering. Schemes like rotary positional embeddings (RoPE) and their extensions let a model generalize to longer sequences, while efficiency tricks — sparse attention (each token attends to a chosen subset instead of everything), sliding-window attention, and smarter memory layouts — cut the quadratic cost enough to make very large windows practical at all. These aren't just research ideas: sparse attention has already shipped in production open-weight models, DeepSeek's among them.

Why bigger isn't always better#

Given all that, it's tempting to treat context size as a pure "more is better" spec. It isn't, for three concrete reasons.

• Effective context is smaller than advertised context. A model that accepts a million tokens does not use all million equally well. Recall tends to sag in the middle — the well-documented "lost in the middle" effect, where facts placed at the very start or end of a long input are retrieved reliably but facts buried in the middle get missed. Vendors publish "needle in a haystack" retrieval scores precisely because raw window size says nothing about whether the model can actually find and use information deep inside it. The usable window is almost always smaller than the number on the box, and it varies by model and by task.
• More context means more cost and latency. Because of the quadratic attention and the growing KV cache, every extra token you stuff in makes the request slower and more expensive — and providers make this explicit. Some 2026 frontier models switch to a higher per-token rate once a prompt crosses a threshold — around 200K tokens for Claude — reflecting the real jump in serving cost. Filling a huge window on every call is a fast way to burn latency and money for little gain.
• Irrelevant context is noise, not signal. A bigger window lets you paste in more — but relevance beats volume. Padding a prompt with marginally related documents can actively hurt: it dilutes the signal, gives the model more chances to latch onto the wrong passage, and can pull the answer off course. A tight, well-chosen 8K-token prompt frequently beats a sloppy 500K-token one on the same question.

Beyond raw size: what actually moves quality#

Window size is one dial among several. Others matter as much or more.

• Retrieval over stuffing. Instead of dumping an entire corpus into the window, retrieval-augmented generation (RAG) fetches only the passages relevant to the current question and puts those in context. It's cheaper, faster, and often more accurate than a giant prompt, because it raises the signal-to-noise ratio the model actually sees. Long context and retrieval are complementary, not rivals: even with a million-token window, you usually want to feed it the right tokens, not all of them.
• Data and training quality. A model trained well on high-quality data reasons better at any context length. A larger window can't compensate for a weaker base model.
• Architecture. Sparse and sliding-window attention, better positional schemes, and mixture-of-experts designs improve how efficiently a model uses the tokens it's given — sometimes more than a bigger window would.
• Hardware and serving. Long context is gated by memory bandwidth and capacity as much as by compute. Newer accelerators and serving tricks — paged KV caches, quantized caches, speculative decoding — are what make million-token requests affordable in the first place.

How to actually pick a window size#

• Match the window to the job. Chat and short Q&A live comfortably in tens of thousands of tokens. Whole-codebase reasoning, long-document analysis, and agents that accumulate a long history are where large windows earn their cost. Don't pay for a million-token model to answer one-line questions.
• Watch effective context, not the headline number. Test whether the model can actually retrieve and use information from the depth you care about, and place the most important material near the start or end of the prompt rather than buried in the middle.
• Curate before you expand. Trim, summarize, or retrieve down to what's relevant before reaching for a bigger window. Better inputs usually beat more inputs.
• Measure quality against cost. Track output quality alongside tokens-per-request. The right window is the smallest one that does the job well — not the biggest one available.

The bottom line#

A context window is the model's working memory: how much it can see at once, shared across your input and its output, and wiped between requests. Its size is bounded by the quadratic cost of attention and the memory footprint of the KV cache, which is exactly why bigger windows are genuinely expensive to build and to run. By mid-2026 the largest windows reach around a million tokens — but that number is a ceiling, not a promise. Effective context is smaller, extra tokens cost real money and latency, and irrelevant context hurts. The models that feel best to use aren't the ones with the largest window; they're the ones that use the tokens they're given well, paired with a workflow that feeds them the right tokens in the first place.

Key Takeaways:

• A context window is a model's shared, per-request working memory — input and output together, measured in tokens — and it's distinct from the knowledge frozen in the model's weights.
• Its limits and costs come from self-attention scaling roughly quadratically with length, plus a KV cache that grows linearly and dominates accelerator memory at long context.
• By mid-2026 several frontier models offer very large context windows — some reaching around a million tokens — but usable ("effective") context is smaller than advertised, and providers may charge a premium once a prompt crosses a threshold (around 200K tokens for some models).
• Bigger isn't automatically better: relevance beats volume, retrieval often beats stuffing, and output quality depends on data, architecture, and hardware as much as on window size.