Latency is the wait between asking a system for something and seeing the result come back. The tool itself may be fine: the distance is long, the server is busy, the model is large, or your request passes through several systems on the way home. With AI you feel it most during inference, the run that turns your input into output. A rate limit is a different problem: the system telling you you’re asking too often.
Picture the DMV scene from Zootopia. Your request is simple, but the line crawls because one part of the process responds slowly. You stand there waiting even though the task should take seconds.
Why you care
Latency changes how you design the job. If an agent waits ten seconds for every tiny step, a hundred-step process gets painful, so you batch work, use a faster model for simple steps, or lean on a cache so a saved answer skips the whole trip. A slow, high-quality run that replaces hours of human work is a fine trade; the same delay kills the experience when a person needs an answer while a client waits.