The real difference between training and inference

People mix up training and inference constantly, and the confusion is not harmless. It is the root of two wrong beliefs: that a chatbot is learning from your conversation as you type, and that running a model is cheap because training is the expensive part. Both are backwards in important ways. The clean line between the two phases is this: training is the only time the model's internal numbers (its weights) change. Inference is using those frozen numbers to produce an answer. Once you hold that distinction, most of the fog clears.

Training: the weights move#

Training is a search problem. You start with a model whose billions of weights are random noise. You show it an example, it produces an output, and you measure how wrong that output was using a loss function. Then you compute, for every single weight, how much it contributed to the error, and you nudge each one a tiny step in the direction that reduces the error. That nudging is backpropagation plus gradient descent. Repeat across trillions of tokens and the weights settle into a configuration that does something useful.

This is enormously expensive, and the reason is structural. Every step requires two passes:

• a forward pass to produce an output
• a backward pass to compute gradients for every weight

The backward pass roughly doubles the compute of the forward pass, and you also have to store activations and optimizer state in memory. That is why training large models takes clusters of accelerators running for weeks. The cost is not the size of the model alone. It is the size times the number of update steps times the bookkeeping that backpropagation demands.

Inference: the weights are frozen#

When you send a prompt to a deployed model, none of that happens. The weights are fixed. The model runs one forward pass, produces a token, and for a chat model feeds that token back in to produce the next. There is no backward pass, no gradient, no learning. The model that answers your question is byte-for-byte identical before and after the exchange.

This is the crux of the "is it learning from me" question. During a single conversation, it is not. The only thing that changes is the text sitting in the context window, which the model re-reads each turn. That can make it feel adaptive within a session, but the moment the session ends, the model has retained nothing. Any real change to its behavior requires a separate training run on collected data, done later, deliberately, by the people who own the model.

A useful analogy, and where it breaks#

Think of training as studying for an exam and inference as sitting the exam. During the exam you cannot rewrite what you know. You can only use it. The notes you are allowed to bring in are the context window: helpful, but not the same as new knowledge.

The analogy breaks in one place worth naming. A student walks out of the exam having learned a little from the experience. The model does not. There is no trickle of learning from inference back into the weights. The two phases are completely separate machinery.

Why this matters for cost and behavior#

The right-hand column is where most operators eventually spend more money. Training is a brutal upfront bill, but you pay it once. Inference is a smaller bill per request that recurs for every user, every prompt, forever. A popular product can spend more on inference in a year than the original training run cost. This is why so much engineering effort now goes into making inference cheaper: quantization, batching, caching, and smaller distilled models that approximate a larger one.

Fine-tuning sits in between#

Fine-tuning muddies the clean split on purpose. It is a short, targeted training run that takes an already-trained model and nudges its weights using a smaller, specialized dataset. It is still training, weights still move, the backward pass is still involved, but it is far cheaper than starting from scratch. The output is a new set of frozen weights that you then serve at inference time like any other.

So the full lifecycle is: train to create the weights, optionally fine-tune to adjust them, then run inference to use them. Learning only ever happens in the first two. Whatever a model says about remembering you, the weights tell the real story, and at inference they do not move.