Inference is the stage in which a trained AI model is run on new inputs to produce outputs, such as answers, classifications or generated text. It is distinct from training, which is when the model learns from data. Every time you send a prompt to a model and receive a reply, inference is taking place.

Inference matters for practical reasons. It determines how fast responses arrive, how much computing power each request uses and how much it costs to operate a service, and these factors influence which model size and hardware are chosen. Providers often charge by the amount of text processed.

Where inference happens also has privacy and compliance implications. Running a model in a provider's cloud means your inputs are sent to that provider, while running it on your own hardware keeps them in your environment. Neither is automatically safe, and the terms and controls differ.

When evaluating a system, measure response time and cost under realistic loads, check where requests are processed and how inputs are stored or logged, and compare these against your requirements for data handling.