Batch API: when it saves money
Where batch API savings come from, when the wait is worth it, and what our batch endpoint does.
What a batch API is
Instead of one request and one immediate answer, you submit many requests at once and the provider runs them when it has spare capacity, usually within a day. Classifying a million support tickets, generating product descriptions overnight and running an evaluation set are typical jobs.
Where the discount comes from
Providers price live traffic for peak demand. Work that can wait fills the quiet hours, so OpenAI, Anthropic and Google each sell their own batch API at around half the normal price, at the time of writing. The saving is real and large, but it comes with conditions: one provider at a time, a turnaround measured in hours, and that provider's own file formats and limits.
What Rapid Router's batch endpoint does
Rapid Router accepts up to 500 chat completions in one call, as inline JSON with no file upload step. They run within a 24 hour window through the same routing and failover as live traffic, and results are kept for 30 days. Each item is billed at the model's standard per-token price: there is no batch discount. A failed item does not sink the batch, and anything unfinished at the deadline is never billed. Full details are in the batch docs.
So our batch endpoint saves engineering, not money: no queue to build, no retry logic, and one call that can mix Claude, GPT, Gemini and open models.
Which to use
A large offline job on a single provider, where a day's wait is fine: the provider's own batch API, for the discount. Mixed models, smaller jobs, or work that needs failover: a gateway batch. Work a user is waiting on: neither; send it live, and look at the other ways to cut cost.
Common questions
How much does a batch API save?
On the providers' own batch APIs, OpenAI, Anthropic and Google each advertise around half the normal price at the time of writing, in exchange for results within about a day. Check each provider's pricing page for the models it covers.
Does Rapid Router's batch API give a discount?
No. Rapid Router batches run at the standard per-token price of each model. They save you building a queue and retry logic, and they work across every model and provider through one call.
How long does a batch take?
On Rapid Router, items run within a 24 hour window; unfinished items when the window closes are marked expired and never billed.
Can I stream inside a batch?
No. Batch items return whole responses, which you collect when the batch completes.