How AI Coding Assistants Actually Work Under the Hood

AI coding assistants are built on a specific stack of machine learning techniques that most developers never see. Understanding that stack changes how you use these tools, where you trust them, and where you don't.

Language Models Trained on Code#

At the core of every coding assistant is a large language model (LLM): a transformer-based neural network trained to predict the next token in a sequence. A tokenizer breaks source code into subword units: for, (, i, in, range might each be a separate token. The model never sees "code" the way you do; it sees integer IDs and probabilities over them.

The line between "code models" and "general models" has mostly dissolved. In 2026 the strongest coding is done by the same frontier general-purpose models that write prose and answer email (Claude Opus 4, GPT-5, Gemini), trained on enormous mixed corpora in which public code, documentation, issue trackers, and technical Q&A are a large and deliberately weighted slice. Code turns out to be a good teacher for reasoning in general, so the labs lean into it. What stays specialized is the fast lane.

Whatever the size, the training signal teaches statistical structure: which tokens tend to follow others in syntactically and semantically coherent code.

The transformer's self-attention mechanism is what makes this work over long spans. Each layer lets every token attend to every other token in view, so the model can track variable scope and dependencies across a file. That's why it "knows" a variable declared at the top of a function is still live fifty lines down.

Context Windows: The Hard Constraint#

When you type, the assistant doesn't see your whole codebase. It sees a context window: a fixed-size slice of tokens. Windows have grown enormously: frontier models in 2026 advertise hundreds of thousands of tokens, and a few reach around a million, enough to hold a small repository at once. But two hard facts survive the growth.

First, real codebases are still bigger than any window: a million tokens is on the order of a hundred thousand lines of code, and mature systems run to millions.

Second, and less obvious: a large window is not the same as reliable recall across all of it.

Attention quality is uneven, and models reliably degrade at finding and using facts buried in the middle of very long contexts: often called "lost in the middle," or context rot. Stuffing everything in is neither free nor equally reliable. What sits near the cursor and near the edges gets used best.

So assistants still curate the window aggressively:

The practical implication is unchanged: bad suggestions often mean the model lacks context, not that it "can't code." Writing a comment describing what you're about to do, or opening a related file, literally changes what the model can see.

Finding the Right Code: Retrieval vs. Agentic Search#

For anything past a single file, the assistant needs a way to decide which parts of a large codebase belong in that limited, best-near-the-edges window. Two approaches dominate in 2026, and the field has actively shifted between them.

Embedding-based retrieval (RAG). The classic approach indexes your codebase ahead of time. Code chunks are converted into embedding vectors (high-dimensional representations produced by a separate, smaller encoder model) so that semantically similar code lands close together in vector space. When you start a function that loads a user from the database, the system embeds your partial code and pulls the nearest neighbors: a similar function elsewhere, a schema definition, a test fixture. Those chunks are prepended to the prompt. The main model never "searches"; it just receives text. The intelligence lives in the retrieval pipeline, and in the chunking, because splitting a function down the middle produces embeddings that miss its intent. Good systems chunk along syntax boundaries, respecting function and class edges.

Agentic search. The newer approach skips the vector index and lets the model navigate the repository the way a developer does: running tools like grep, listing directories, and reading the files it decides are relevant, in a loop.

Many production tools now blend both (a quick semantic pass to locate candidates, then tool-driven reading to confirm), but the direction of travel has been toward letting a capable model simply look, rather than pre-digesting the codebase for it.

The Inference Pipeline#

When you pause typing, roughly this happens:

A few mechanisms make this fast enough to feel live. KV caching stores the intermediate attention state for tokens already processed, so each new token doesn't recompute the entire context from scratch. Prompt caching extends that idea across requests, so a large unchanging preamble (your system prompt, stable context) is computed and billed once and reused. Weights are frequently quantized, reduced from 16- or 32-bit floats to 8- or 4-bit integers, which shrinks memory and speeds the math at a modest accuracy cost, especially for the small autocomplete models that must answer in well under a second.

All of it runs on GPUs or dedicated inference accelerators, usually in a datacenter, occasionally on-device for the smallest models. That sub-second latency target for inline completion is the reason those models stay small and quantized; the slower, larger frontier models are reserved for work where you'll happily wait a few seconds for a better answer.

Fine-Tuning: From Raw Model to Assistant#

A base model trained purely to predict code is powerful but raw. Ask it to "write a unit test for this function" and it may just continue the text rather than do the task. Three stages turn it into an assistant.

Training against an executable signal is a large part of why recent models got sharply better at producing code that compiles and passes tests. The catch: that verification happens during training, on training problems. At inference, on your problem, the model is back to prediction with no test runner attached, unless the surrounding tool provides one.

Some providers also do domain-specific fine-tuning on a customer's own codebase (internal APIs, naming conventions, architectural patterns) which is expensive but meaningfully lifts relevance for large teams with idiosyncratic code.

Reasoning and the Agentic Loop#

Two shifts turned these tools from autocomplete into something closer to a collaborator.

Reasoning models are trained to generate a long internal chain of thought before committing to an answer: spending extra inference compute to plan, consider cases, and check themselves. For coding, where a bug can hide in a single branch, that deliberate step measurably improves results, especially on multi-step problems. The cost is latency and tokens, which is exactly why tools split work between a fast model for keystrokes and a reasoning model for hard tasks.

The agentic loop wraps the model in a control loop with tools and, crucially, feedback. Rather than emit one block of code and stop, the assistant plans, edits files, runs the build or the tests, reads the errors, and tries again: iterating until things pass or it gives up. This is how most serious AI-assisted work is done in 2026 (Claude Code, Cursor's agent, Codex, and similar). It is the closest these systems come to closing the gap between generation and verification: the compiler and test runner become the ground truth the base model lacks. It doesn't grant the model understanding (each step is still statistical prediction), but a loop that can see its own failures and react is meaningfully more reliable than one-shot generation.

The Structural Blind Spots#

Underneath the reasoning and the loops, the engine still predicts tokens from learned patterns.

On its own, outside a tool that actually runs things, it has no runtime, no type checker, no linter, no test runner. Nothing verifies that the code works.

It produces code that looks like correct code because that is what the training data mostly contained. That yields predictable failure modes:

Reasoning narrows these; the agentic loop narrows them further, because running the code catches a hallucinated API or a failing test that pure generation would happily ship. But none of it installs genuine understanding of correctness, only more chances to notice when the statistics led somewhere that doesn't run.

Where This Leaves You#

An AI coding assistant is a sophisticated pattern-completion engine (transformers, heavily code-weighted training, curated context, tuned inference) increasingly wrapped in reasoning and a tool-driven loop. It works because code is highly structured and its patterns are strong enough to make prediction surprisingly accurate. It fails in structured ways because it has no ground truth beyond training, except what a surrounding tool hands it.

A few things worth keeping in mind:

Know the mechanism and you know where to lean on these tools hard, and where to stay skeptical.