Latency

24Latency and performance

Rapid Router adds one network hop between you and the provider. The routing itself is a lookup and a forward, so the number that dominates your response time is the provider's own speed, not ours.

Four things actually move the needle, roughly in order of size:

• Stream. Set stream: true. A 900 token answer takes seconds to finish but the first token lands in a fraction of that, and a user reading along does not feel the rest.
• Pick for speed, not size. A smaller model on fast silicon routinely beats a large model on ordinary hardware by several times, and for extraction, classification and routine drafting the answer is the same.
• Keep the connection alive. Reuse one client instead of constructing a new one per call. A fresh TLS handshake costs tens of milliseconds every time.
• Cap your output. max_tokens bounds the worst case. Without it a chatty model decides your latency for you.

Live latency per provider is on the providers page, measured from real traffic rather than a synthetic probe.

A provider that fails is routed around where the model has a second route. See failover for when that applies and when the error is returned to you instead.