How AI Coding Assistants Actually Work Under the Hood

AI coding assistants are built on a specific stack of machine learning techniques that most developers never see. Understanding that stack changes how you use these tools — where you trust them, and where you don't.

Language Models Trained on Code#

At the core of every coding assistant is a large language model (LLM) — a transformer-based neural network trained to predict the next token in a sequence. A tokenizer breaks source code into subword units: for, (, i, in, range might each be a separate token. The model never sees "code" the way you do; it sees integer IDs and probabilities over them.

The line between "code models" and "general models" has mostly dissolved. In 2026 the strongest coding is done by the same frontier general-purpose models that write prose and answer email — Claude Opus 4, GPT-5, Gemini — trained on enormous mixed corpora in which public code, documentation, issue trackers, and technical Q&A are a large and deliberately weighted slice. Code turns out to be a good teacher for reasoning in general, so the labs lean into it. What stays specialized is the fast lane: the low-latency inline completion you get as you type is often served by a smaller, cheaper model, while the heavy work — multi-file edits, debugging, "explain this" — routes to a big frontier model.

Whatever the size, the training signal teaches statistical structure: which tokens tend to follow others in syntactically and semantically coherent code.

The transformer's self-attention mechanism is what makes this work over long spans. Each layer lets every token attend to every other token in view, so the model can track variable scope and dependencies across a file. That's why it "knows" a variable declared at the top of a function is still live fifty lines down.

Context Windows: The Hard Constraint#

When you type, the assistant doesn't see your whole codebase. It sees a context window — a fixed-size slice of tokens. Windows have grown enormously: frontier models in 2026 advertise hundreds of thousands of tokens, and a few reach around a million, enough to hold a small repository at once. But two hard facts survive the growth.

First, real codebases are still bigger than any window — a million tokens is on the order of a hundred thousand lines of code, and mature systems run to millions. Second, a large window is not the same as reliable recall across all of it. Attention quality is uneven; models reliably degrade at finding and using facts buried in the middle of very long contexts — often called "lost in the middle," or context rot. Stuffing everything in is neither free nor equally reliable. What sits near the cursor and near the edges gets used best.

So assistants still curate the window aggressively:

• Cursor-position heuristics: Code around your cursor gets priority. Many completion models use a fill-in-the-middle (FIM) format — prefix before the cursor, suffix after — so the suggestion fits what comes next, not just what came before.
• Import-graph analysis: If your file imports from utils/auth.ts, the assistant pulls that file's relevant exports in as context.
• Recently opened files: Editors include snippets from what you've had open.
• Truncation strategies: When space runs out, content gets dropped — usually from the middles of files, preserving the edges where signatures and structure cluster.

The practical implication is unchanged: bad suggestions often mean the model lacks context, not that it "can't code." Writing a comment describing what you're about to do, or opening a related file, literally changes what the model can see.

Finding the Right Code: Retrieval vs. Agentic Search#

For anything past a single file, the assistant needs a way to decide which parts of a large codebase belong in that limited, best-near-the-edges window. Two approaches dominate in 2026, and the field has actively shifted between them.

Embedding-based retrieval (RAG). The classic approach indexes your codebase ahead of time. Code chunks are converted into embedding vectors — high-dimensional representations produced by a separate, smaller encoder model — so that semantically similar code lands close together in vector space. When you start a function that loads a user from the database, the system embeds your partial code and pulls the nearest neighbors: a similar function elsewhere, a schema definition, a test fixture. Those chunks are prepended to the prompt. The main model never "searches"; it just receives text. The intelligence lives in the retrieval pipeline — and in the chunking, because splitting a function down the middle produces embeddings that miss its intent. Good systems chunk along syntax boundaries, respecting function and class edges. Cursor, for example, still builds a custom embedding index of your repo.

Agentic search. The newer approach skips the vector index and lets the model navigate the repository the way a developer does — running tools like grep, listing directories, and reading the files it decides are relevant, in a loop. Anthropic's Claude Code notably dropped embedding-based indexing in favor of this tool-driven search, and other agent tools followed. The tradeoffs are real and opposite. Agentic search is exact and never stale — it reads the actual current files, with no index to rebuild — but it burns more model turns and tokens to find things a single semantic query might have surfaced at once. Embeddings give fast, fuzzy, semantic recall, but the index can drift out of date, and a similarity match can't reason its way to the right file the way a guided search can.

Many production tools now blend both — a quick semantic pass to locate candidates, then tool-driven reading to confirm — but the direction of travel has been toward letting a capable model simply look, rather than pre-digesting the codebase for it.

The Inference Pipeline#

When you pause typing, roughly this happens:

1. Prompt construction: The assistant assembles the window — cursor prefix, suffix, retrieved or read snippets, system instructions — into one formatted prompt.
2. Tokenization: Text becomes integer IDs.
3. Forward pass: The IDs flow through the transformer's layers, each applying attention and feed-forward transformations. The output is a probability distribution over the whole vocabulary — the model's guess at the next token.
4. Sampling: The model picks a token — the highest-probability one (greedy) or one drawn from the top candidates (temperature, nucleus/top-p sampling). Code assistants tend toward low temperature or greedy decoding, because for code, correctness beats variety.
5. Autoregressive generation: The chosen token is appended and fed back in, and the process repeats until a stop condition — an end-of-sequence token, a newline, a length cap.

A few mechanisms make this fast enough to feel live. KV caching stores the intermediate attention state for tokens already processed, so each new token doesn't recompute the entire context from scratch. Prompt caching extends that idea across requests, so a large unchanging preamble — your system prompt, stable context — is computed and billed once and reused. Weights are frequently quantized, reduced from 16- or 32-bit floats to 8- or 4-bit integers, which shrinks memory and speeds the math at a modest accuracy cost — especially for the small autocomplete models that must answer in well under a second.

All of it runs on GPUs or dedicated inference accelerators, usually in a datacenter, occasionally on-device for the smallest models. That sub-second latency target for inline completion is the reason those models stay small and quantized; the slower, larger frontier models are reserved for work where you'll happily wait a few seconds for a better answer.

Fine-Tuning: From Raw Model to Assistant#

A base model trained purely to predict code is powerful but raw. Ask it to "write a unit test for this function" and it may just continue the text rather than do the task. Three stages turn it into an assistant.

Supervised fine-tuning (SFT) trains on curated instruction-response pairs — a request paired with an ideal response — teaching the model the format and intent of following instructions.

Reinforcement learning from human feedback (RLHF) goes further: human raters rank responses, a reward model learns to predict those rankings, and the model is optimized to score well. This sharpens instruction-following, tone, edge-case handling, and refusal of harmful requests.

Reinforcement learning from verifiable rewards is the piece that matters most for code, and it's where these tools quietly get a taste of ground truth. Instead of a human judging the answer, the generated code is run — against a test suite, a compiler, a linter — and the model is rewarded when it actually passes. Training against an executable signal is a large part of why recent models got sharply better at producing code that compiles and passes tests. The catch: that verification happens during training, on training problems. At inference, on your problem, the model is back to prediction with no test runner attached — unless the surrounding tool provides one.

Some providers also do domain-specific fine-tuning on a customer's own codebase — internal APIs, naming conventions, architectural patterns — which is expensive but meaningfully lifts relevance for large teams with idiosyncratic code.

Reasoning and the Agentic Loop#

Two shifts turned these tools from autocomplete into something closer to a collaborator.

Reasoning models are trained to generate a long internal chain of thought before committing to an answer — spending extra inference compute to plan, consider cases, and check themselves. For coding, where a bug can hide in a single branch, that deliberate step measurably improves results, especially on multi-step problems. The cost is latency and tokens, which is exactly why tools split work between a fast model for keystrokes and a reasoning model for hard tasks.

The agentic loop wraps the model in a control loop with tools and, crucially, feedback. Rather than emit one block of code and stop, the assistant plans, edits files, runs the build or the tests, reads the errors, and tries again — iterating until things pass or it gives up. This is how most serious AI-assisted work is done in 2026 (Claude Code, Cursor's agent, Codex, and similar). It is the closest these systems come to closing the gap between generation and verification: the compiler and test runner become the ground truth the base model lacks. It doesn't grant the model understanding — each step is still statistical prediction — but a loop that can see its own failures and react is meaningfully more reliable than one-shot generation.

The Structural Blind Spots#

Underneath the reasoning and the loops, the engine still predicts tokens from learned patterns. On its own — outside a tool that actually runs things — it has no runtime, no type checker, no linter, no test runner. Nothing verifies that the code works. It produces code that looks like correct code because that is what the training data mostly contained. That yields predictable failure modes:

• Hallucinated APIs: Calls to methods that don't exist, or existed only in another version. The shape of the pattern is right; the specific name is invented.
• Security antipatterns: If insecure code was common in training data — SQL built by string concatenation, weak randomness — the model reproduces it just as fluently as good code.
• Stale knowledge: Training has a cutoff. A library that changed its API afterward will still be called the old way, confidently. (Retrieval and agentic search can partly counter this by pulling in the current code and docs.)
• Context blindness: Code that's locally correct can be globally wrong if it contradicts something outside the window the model never saw.

Reasoning narrows these; the agentic loop narrows them further, because running the code catches a hallucinated API or a failing test that pure generation would happily ship. But none of it installs genuine understanding of correctness — only more chances to notice when the statistics led somewhere that doesn't run.

Where This Leaves You#

An AI coding assistant is a sophisticated pattern-completion engine — transformers, heavily code-weighted training, curated context, tuned inference — increasingly wrapped in reasoning and a tool-driven loop. It works because code is highly structured and its patterns are strong enough to make prediction surprisingly accurate. It fails in structured ways because it has no ground truth beyond training, except what a surrounding tool hands it.

A few things worth keeping in mind:

• The model only acts on what reaches its context window; retrieval and agentic search extend its reach, and their quality caps the quality of the output.
• Suggestions are sampled from a probability distribution, not looked up as facts.
• The failure modes are structural, not bugs waiting to be patched.
• Loops that actually run and test the generated code are far more trustworthy than raw generation — prefer them for anything that matters.

Know the mechanism and you know where to lean on these tools hard, and where to stay skeptical.