AWS published a benchmark on 8 September comparing three generations of its GPU instances on 30 billion parameter mixture-of-experts models. Its newest family, G7, built on NVIDIA's Blackwell RTX PRO 4500, came out ahead on throughput, on latency and on cost per token.
A company benchmarking its own newest hardware and winning is not a story. What is underneath the result is, because two of the numbers cut against the way most teams size an inference machine.
The box that won had half the GPUs#
The first test runs Qwen3-Coder-30B through the SageMaker DJL Large Model Inference container on three instances of the same 12xlarge size.
The G5 and G6 instances each bring four GPUs and 96 GB of aggregate GPU memory. The G7 instance brings two GPUs and 64 GB. Half the accelerators, two thirds of the memory.
The two-GPU machine won anyway. AWS reports G7 at 391.3 output tokens per second, approximately 60.8 per cent above G6 and 13.0 per cent above G5. Average request latency fell by approximately 37.6 per cent against G6 and 10.8 per cent against G5. At the ninety-ninth percentile, where the slowest requests live and where a user actually notices, those gaps widen to 54.7 per cent and 20.2 per cent.
AWS gives the mechanism in its own words: "MoE architectures are memory-bandwidth bound during token generation (decoding) because each token activates only a small subset of experts."
That sentence is the transferable part of the whole post. A mixture-of-experts model routes each token through a small fraction of its weights, so the speed of generation tracks how fast memory can be read, not how much of it there is. Once the model fits, extra capacity buys nothing for decoding. Bandwidth keeps paying.
Which is why the amount of memory is only the first question when you are sizing for a model, and never the whole one. Memory speed, not memory size, is what a decode step waits on, and it is a large part of why a GPU suits this work better than a CPU in the first place.
There is a second mechanism, and AWS is specific about it: only G7 has native FP4 tensor core support. G5 and G6 can run the same 4-bit NVFP4 weights, but without hardware acceleration for the format.
The cheapest instance and the fastest instance are not the same machine#
The second test is a different question asked a different way. It runs NVIDIA's Nemotron-3-Nano-30B-A3B-NVFP4 through SageMaker's inference recommendation workflow with vLLM, and asks not which instance is quickest but which configuration to deploy.
For a chat-shaped workload of 512 input tokens and 256 output tokens, AWS reports g7.48xlarge highest on throughput at 2,397 output tokens per second, and g7.2xlarge lowest on cost at $0.90 per one million output tokens.
Those are two different instances, and neither of them is "the best G7". Ask the recommendation engine to optimise for throughput and it hands back the 48xlarge. Optimise for unit cost and you deploy the 2xlarge and accept less capacity from it. AWS is explicit that throughput was the target it set and that cost was derived afterwards, which is the honest way round to describe it.
The lesson survives the vendor. Any question of the form "which one should I run" is unanswerable until somebody says whether the thing being minimised is latency, cost per token, or total tokens per hour. Those three have different winners here, on the same model, in the same instance family, on the same day.
The same model, 2.5 times the cost per token#
Now hold the model and the family still and change only the shape of the traffic. For a retrieval-shaped run of 3,500 input tokens and 200 output tokens, AWS reports the cheapest configuration at $2.29 per one million output tokens, against $0.90 for the chat shape.
Same model. Same instance family. The prompt got longer, and the cost of producing one token went up roughly two and a half times, because the machine now spends much more of its time reading the question and much less writing the answer.
Anyone who prices a retrieval product off a chat benchmark is budgeting the wrong number, and the error is not small. This is the ordinary shape of what inference actually costs, and it stays invisible for as long as you only ever price the model rather than the workload.
Against the older families, AWS reports its cheapest G7 configuration at approximately 1.4 times lower cost per token than G6e on both workload shapes, and approximately 4.9 times lower than G6 on the chat shape and 5.6 times lower on the retrieval one.
The cost numbers are derived, and the input is not on the page#
AWS shows its working, which is more than most benchmarks do: cost per one million output tokens is the hourly instance rate multiplied by a million, divided by output tokens per second multiplied by 3,600.
The throughput half of that is measured and published. The price half is not. The post names no hourly rate for any instance it prices, and the SageMaker pricing page builds its rate table from a live feed rather than writing it into the page. So the two dollar figures above cannot be reproduced from the post on its own.
That is not a hidden number, and it is not a criticism. It is a number you have to go and fetch, and it is also the one most likely to be different for you, because rates move with region and with whatever commitment you have signed. The formula is the genuinely useful part of that section, precisely because it is the part you can re-run against the rate you actually pay.
What this establishes, and what it does not#
Two limits, both visible in the post itself.
The two use cases are not one experiment. They use different models, different serving stacks, and different instance sizes. AWS presents them as two complementary methods rather than one controlled comparison, which is a fair description, but it does mean the G7 result rests on two narrow tests rather than one broad one.
And this is a vendor benchmarking its own newest instance family, with its own tooling, on workloads it picked. AWS discloses the models, the GPU counts, the memory totals, the workload shapes and the arithmetic. That is a good deal more disclosure than most vendor benchmarks carry. It is still a vendor benchmark, and the number to wait for is the first one produced by somebody who is not selling the instance.
What does transfer, whoever runs it next, is the mechanism: for mixture-of-experts decoding, bandwidth beats capacity, and the shape of your traffic moves your cost per token further than the choice of instance does.
Discussion