Guides
Guides
Task-by-task guides: streaming, tools, structured output, media, batches, routing, budgets, errors and limits.
Build
StreamingServer-sent events in the standard chunk shape, ending with a usage chunk that carries the billed cost.Tool callingLet a model call your functions using the OpenAI tools shape, translated for Anthropic models.Structured outputJSON mode and JSON schema through response_format, emulated on Anthropic models.Response healingRepair almost-JSON answers (fences, trailing commas, truncation) without a second model call.Reasoning modelsHow thinking tokens are billed and reported, and the max_tokens setting that leaves no room to answer.PDF, audio and video inputAttach PDFs, audio and video to chat messages; the same request body works on every model.Web searchGround any model in live web results with the web plugin or the :online suffix, with citations.
Images, audio and batches
Images, speech and videoGenerate and edit images, text to speech, transcription, translation and video, billed per unit.Voice pipelineAudio in, spoken answer out: transcription, a chat model and speech in one call.Batch APIQueue up to 500 chat completions in one call; they run within 24 hours and results are kept 30 days.
Routing and fallbacks
Failover and routingHow a failing provider falls over to the next route, and the provider object that steers the choice.Auto routerrapidrouter/auto picks the model per prompt: light, code, Indic or heavy.PresetsNamed, versioned request templates you change in the dashboard without a deploy.Message transformsmiddle-out compression for conversations longer than the model can take.Latency and performanceThe four things that actually move response time, roughly in order of size.
Cost and budgets
BudgetsCap spend per key, per account and per workspace, and the error a request gets once a cap is hit.Spend alertsEmail, WhatsApp or webhook alerts on daily and monthly thresholds, plus automatic anomaly alerts.Bring your own keysRoute requests on your own provider keys and pay only the platform fee.Prompt cachingCached input billed at the provider cache rate, reported on every response.Response cachingOpt-in replay of an identical request, billed at zero.Free tierWhich models are free, the daily limits per account, and the 429 when they run out.
Errors and limits
ErrorsStatus codes, the one error envelope, and the health endpoint worth monitoring.Rate limitsRequests per minute per key and requests in flight per account, both growing with your balance.IdempotencyRetry a request whose response was lost without paying twice, with the Idempotency-Key header.Tracingx-request-id and W3C traceparent on every response, and OTLP export for self-hosted gateways.
Apps and teams
Data and privacy
Data collectionWhat is recorded per request, and why prompts and responses are not stored unless you turn on capture.Provider loggingWhere a prompt goes after the gateway: links to each provider's own policy.Request captureOpt-in, per-key storage of request and response bodies for debugging, deleted on a schedule.