Inference is the stage in which a trained model is put to work. Training adjusts the weights, and inference uses them: you give the model an input, and it produces an output, such as the next tokens of an answer, a classification or an image. Every time someone sends a prompt to an AI product, inference runs.
For a deployed product it is usually the larger and more continuous cost, because training happens once while inference happens with every request, and agents and long reasoning multiply it. It is shaped by the size of the model, the length of the input and output, the hardware, the batching of requests and the precision used.
The main measures are latency, meaning how quickly a user sees a response, throughput, meaning how much work the system completes over time, and cost per token. They pull against one another, and improving one often costs another.
Choose the smallest model that meets your quality needs, measure latency and cost on realistic traffic, and look at optimization options such as quantization, caching and batching.