Prompt caching

14Prompt caching

When you send the same long prefix repeatedly, a system prompt, a document, a set of examples, some providers keep the processed form of it and charge far less to read it back. Rapid Router passes that through and bills cached input at the provider's cache rate rather than the full input rate.

You do not configure anything. If the provider reports cached tokens, they are priced as cached. Where we hold a cache rate it currently runs 80% to 92% below the input rate. Where we do not hold one, cached tokens bill at the normal input price, so an unknown rate never costs you more than it did before.

Every response tells you what happened. usage.prompt_tokens_details.cached_tokens is how much of your prompt was served from cache, and usage.cost_without_cache appears only when a request actually saved money, so you can see the difference per call rather than guess at it monthly.

"usage": {
  "prompt_tokens": 12480,
  "completion_tokens": 210,
  "prompt_tokens_details": { "cached_tokens": 11900 },
  "cost": 0.000452,
  "cost_without_cache": 0.001361
}

To get hits, keep the stable part of your prompt first and the variable part last. Caching keys on the prefix, so anything that changes early invalidates everything after it.