Latency is the elapsed time between an action and the result the system returns. For an AI service, it may include network travel, request queuing, token processing, model execution and any retrieval or tool calls. A response time measured at a GPU is not the same as the delay experienced by a user.
Location matters because signals take time to cross networks. Processing data nearer a sensor or device can cut round trips to a distant data center, particularly when a task needs several exchanges. But a small local processor may take longer to run the model, so edge computing is not automatically the lowest-latency option.
Latency changes with load. A lightly used system may respond quickly and still develop long queues at peak demand. For generated text, teams often separate time to first token from the time needed to finish the whole answer, since these reflect different parts of the user experience.
Measure latency end to end under realistic traffic and report a distribution, not only one average. Compare median and high-percentile response times, identify bottlenecks and include network, inference and application overhead before selecting a deployment location.