The real difference between training and inference

People mix up training and inference constantly, and the confusion is not harmless. It is the root of two wrong beliefs: that a chatbot is learning from your conversation as you type, and that running a model is cheap because training is the expensive part. Both are backwards in important ways. The clean line between the two phases is this: training is the only time the model's internal numbers (its weights) change. Inference is using those frozen numbers to produce an answer. Once you hold that distinction, most of the fog clears.

Training: the weights move#

Training is a search problem. You start with a model whose billions of weights are random noise. You show it an example, it produces an output, and you measure how wrong that output was using a loss function. Then you compute, for every single weight, how much it contributed to the error, and you nudge each one a tiny step in the direction that reduces the error. That nudging is backpropagation plus gradient descent. Repeat across trillions of tokens and the weights settle into a configuration that does something useful.

This is enormously expensive, and the reason is structural. Every step requires two passes:

• a forward pass to produce an output
• a backward pass to compute gradients for every weight

The backward pass roughly doubles the compute of the forward pass, and you also have to store activations and optimizer state in memory. That is why training large models takes clusters of accelerators running for weeks. The cost is not the size of the model alone. It is the size times the number of update steps times the bookkeeping that backpropagation demands.

Inference: the weights are frozen#