Inference Throughput — Glossary

The longer version

It counts how many tokens a system generates, or processes, in total each second across all users. It is different from latency, which is how long one request takes. Batching many requests together raises throughput, since the hardware is used better, but can make each individual request wait longer.

It sets the cost of serving a model. The more tokens per second a GPU can produce, the lower the cost per token. Quantization, a faster serving engine and hardware with more memory bandwidth all increase it.

Published figures depend on the test: prompt length, output length, batch size and precision all change the result.

Compare numbers only when the conditions match, measure with your own prompts and report throughput together with latency so that the trade-off is visible.

Where we use “Inference Throughput”

CybersecurityMistral Large 4 Goes Open-Weight: What the Independent Scores Support
Mistral says its open-weight Large 4 leads in defensive security, but the independent Cyber Index puts it third. Here is what the scores, the hardware needs and the dual-use risk actually support.
Nov 1, 2026

The longer version

Where we use “Inference Throughput”

Get the Core in your inbox.