Mixture-of-Experts Models: How They Work and Why They Cut Inference Costs

Mixture-of-experts (MoE) is the architecture behind most of the frontier open-weight models shipping in 2026: DeepSeek-V3, Qwen3, Llama 4, OpenAI's gpt-oss, and Mistral's Mixtral line before them. The core promise is simple: build a model with a very large total parameter count, but only activate a small slice of it for any given token. You get something close to the capacity of a huge model at roughly the per-token compute cost of a small one.

DeepSeek-V3 is a clean reference point: 671 billion total parameters, but only about 37 billion active per token. OpenAI's gpt-oss-120b is more extreme, roughly 117 billion total with around 5 billion active, so under 5% of the model fires on any given forward pass. That gap between total and active parameters is the whole game.

The dense baseline, and its problem#

To understand why MoE matters, start with how a standard dense transformer works. Every token you feed into a dense model activates every parameter in every layer. If the model has 70 billion parameters, roughly all 70 billion participate in processing each token. That is computationally expensive, and the cost scales linearly with model size: double the parameters, double the FLOPs per token.

Dense models are also wasteful in a conceptual sense. Not every token needs every kind of capacity. A token in the middle of a protein-folding discussion does not particularly need whatever the model uses for rhyme or legal boilerplate. A dense model applies all of its capacity to every token anyway, because it has no mechanism to do otherwise. MoE is, at heart, an attempt to give the model that mechanism.

Routers and experts: the MoE mechanism#

MoE models replace some or all of the feed-forward network sub-layers in a transformer with an expert layer. Each expert layer contains many independent feed-forward sub-networks (the experts) plus a small router, also called a gating network, that decides which experts handle each token. Per token, the expert layer works like this:

1. The router reads the token's current hidden state.
2. A small linear layer produces one score (a logit) for each expert.
3. The top-K experts are selected by score. K is small, often 1 or 2, but 8 in DeepSeek-V3, chosen from a pool that ranges from 8 in Mixtral to 128 in Llama 4 Maverick to 256 in DeepSeek-V3.
4. Only those K experts process the token. Their outputs are combined as a weighted sum, where the weights come from normalizing the router scores.

Crucially, not everything is sparse. Attention layers, embeddings, and in many modern designs one or more always-on shared experts run for every token. So the active parameter count is this always-on backbone plus the K routed experts you selected. One honest caveat, because the expert framing invites it: learned experts rarely map onto tidy human categories like the biology expert or the code expert. When researchers inspect routing, it correlates more with lower-level features (token identity, syntax, surface patterns) than with clean semantic domains. The specialist metaphor is a useful intuition for why sparsity can work; it is not a literal description of what each expert knows.

Why this makes inference cheaper#

Compute cost in neural network inference is dominated by floating-point operations, and FLOPs scale with the parameters actually used per forward pass, not the parameters that merely exist. By keeping active parameters well below total parameters, MoE cuts the FLOPs per token in direct proportion. The practical consequence: you can serve a higher-capacity model for roughly the per-token compute cost of a much smaller dense one. The dormant experts come nearly free at inference, because most of the time they simply do not run. But there is a subtler point that most explanations get wrong, and it cuts in two directions at once:

At training time the savings are meaningful but more nuanced. You still have to route enough tokens to each expert to train it, the routing itself adds overhead, and load-balancing machinery complicates the loss. But for a given quality target, total training FLOPs are typically lower than for an equivalent-quality dense model, which is exactly what the Switch Transformer work demonstrated and what has held up at scale since.

The make-or-break problem: load balancing#

MoE's central engineering headache is load balancing. If the router keeps sending most tokens to the same few experts, those experts overtrain while the rest atrophy, a failure mode called expert collapse. You end up paying to store a huge model that effectively behaves like a small dense one, having thrown away the entire point of the architecture.

The classic fix is an auxiliary load-balancing loss added during training, which penalizes uneven traffic across experts. The tension is that this term fights the main objective: pushing tokens toward balance can pull them away from the expert that would have handled them best. A newer approach, popularized by DeepSeek-V3, is auxiliary-loss-free balancing: each expert gets a learned bias added to its routing score, nudged down when it is overloaded and up when it is idle, so balance emerges without a competing gradient dragging on quality. This is a notable part of why the recent large open MoE models train cleanly.

The real infrastructure cost: distributed deployment#

Here is what the cheaper-inference framing tends to understate: serving a large MoE model means holding every expert's weights in memory at once, even though only a handful run per token. A 671B-parameter model needs the aggregate memory to store 671B parameters, spread across a serving cluster, regardless of how lean the per-token compute is. Because the experts do not fit on one accelerator, MoE inference typically demands expert parallelism, with different experts living on different GPUs or TPUs.

When a token's router selects experts spread across several devices, the system has to shuttle activations between them, producing an all-to-all communication pattern that can dominate latency if the interconnect is slow. On rack-scale systems that link dozens of accelerators with high-bandwidth interconnects (NVIDIA's NVLink domains on Blackwell-class hardware, or Google's TPU pods) the shuffle stays cheap and the compute savings show up as real dollar savings. On a smaller setup with ordinary networking between nodes, that same shuffle can eat much of the theoretical win. MoE is genuinely cheaper per token in compute terms, but it buys that with infrastructure complexity.

Fine-tuning is trickier than dense#

• Routing drift: fine-tuning on a narrow domain can push the router to over-index on a handful of experts for that domain's tokens, degrading general behavior as the routing distribution shifts.
• All experts need gradients: even though only a few activate per token, training touches all of them across a batch, so fine-tuning memory is closer to the full model than the active slice, a nasty surprise if you budgeted from inference cost.
• Adapters and routers: methods like LoRA apply cleanly to the expert feed-forward networks, but there is no settled answer on whether to adapt the router itself. Freeze it and you may keep misrouting domain tokens; train it and you risk routing drift.

Where the architecture is headed is fairly clear: many small routed experts plus one or more always-on shared experts as the default recipe, bias-based balancing that removes the quality-fighting auxiliary loss, hardware co-designed so the all-to-all traffic is cheap, and hybrid stacks that keep some layers dense and make only others MoE, giving designers a dial between full density and full sparsity.