Prompt caching

Prompt caching

Some providers cache the beginning of a prompt and bill the repeated part at a lower rate.
On a long agent session that reuses the same system prompt and file contents, this is the
difference between a cheap run and an expensive one.

Which models cache

capabilities.prompt_cache on the model row:

curl -s https://hisoka.ai/api/v1/models \
  | jq '.data[] | select(.capabilities.prompt_cache) | .id'

A model without the flag bills the whole prompt at the input rate. There is no penalty for
sending a long prompt either way, only a missed discount.

How it shows up

Your bill reports the cached portion separately, so the discount is visible rather than
implied:

{
  "type": "usage",
  "amount_micro_usd": -26,
  "model": "glm-5.3-flash",
  "input_tokens": 502,
  "cached_tokens": 0,
  "output_tokens": 5
}

On a call that hit the cache, cached_tokens is a large share of input_tokens, and the
charge is far below what the same token count would cost fresh:

fresh_input  = input_tokens - cached_tokens
cached_input = cached_tokens

price = fresh_input  / 1,000,000 * input_rate
      + cached_input / 1,000,000 * cached_rate
      + output       / 1,000,000 * output_rate

cached_rate is lower than input_rate on every model that supports caching.

What makes a cache hit

Caching matches a prefix. The beginning of your prompt must be byte-identical to the
previous request for the provider to reuse it.

Practical consequences:

• Put the stable part first. System prompt, then tool schemas, then history, then the
  new turn. Reordering the front of a prompt defeats the cache.
• Do not edit the system prompt every request. A timestamp or a request id inside it
  makes every prompt unique.
• A long session benefits most. A one-shot request has nothing to reuse.

It is not something you enable

There is no parameter to turn on. The provider decides, the gateway reports what it was
billed, and cached_tokens tells you whether it happened. If you expect a hit and
cached_tokens stays 0, the prefix is changing between calls.

Why the discount is worth chasing

A coding agent re-sending a 50,000-token transcript each turn pays for that transcript
every turn. With caching, the repeated portion bills at the cached rate, which is where
most of the saving in a long session comes from.

Watch it per model with the summary endpoint:

curl -s "https://hisoka.ai/api/v1/usage/summary?period=7d" \
  -H "Authorization: Bearer $HISOKA_API_KEY" \
  | jq '{tokens: .tokens, usd: .usd, per_1k: .usd_per_1k_tokens}'

A low usd_per_1k_tokens on a session that sends a lot of input is the signature of a
working cache.