Rate limits

Rate limits

Two different things share the 429 status, and confusing them leads to the wrong fix.

Your own route limit

Each route has a per-key limit. Exceeding it answers:

{
  "error": {
    "code": "gateway_rate_limited",
    "message": "Rate limit exceeded (gateway_rate_limited)"
  }
}

This is about how fast you are calling. Backing off fixes it.

Upstream throttling

A 429 rate_limited means a provider throttled the request. The Retry-After header and
the message both carry the wait, and x-should-retry is true.

A 429 pool_exhausted means every account behind that model is throttled at once. The
message names the soonest cooldown and carries the same Retry-After.

pool_exhausted is a wait, not a reason to switch models

The model is fine. Switching costs you a model choice, and often a worse one, for a wait
you could have sat through. Read Retry-After and retry the same request.

Reading the difference

Signal Meaning
code: gateway_rate_limited you are calling too fast; slow down
code: rate_limited one provider is throttling; wait and retry
code: pool_exhausted every provider account is throttled; wait and retry
x-should-retry: true the gateway believes a retry can succeed
Retry-After seconds to wait, when it is known

Retry with backoff, not in a tight loop

A tight retry loop against a 429 costs you nothing in money and everything in
throughput: it keeps the limit saturated for as long as you keep calling. Respect
Retry-After when it is present, and use exponential backoff with jitter when it is not.

import random, time, httpx

def call_with_retry(payload, key, attempts=4):
    for n in range(attempts):
        r = httpx.post(
            "https://hisoka.ai/api/v1/chat/completions",
            headers={"Authorization": f"Bearer {key}"},
            json=payload,
            timeout=120,
        )
        if r.status_code < 400:
            return r.json()

        body = r.json().get("error", {})
        if r.headers.get("x-should-retry") != "true":
            raise RuntimeError(f"{body.get('code')}: {body.get('message')}")

        wait = float(r.headers.get("Retry-After") or 0) or (2 ** n + random.random())
        time.sleep(min(wait, 30))

    raise RuntimeError("retries exhausted")

Bursts are handled upstream

The gateway spreads a burst across the accounts behind a model rather than sending
everything to the first one, and an account that fails is benched briefly so a client
burst does not all land on a broken connection. Most short blips never reach you.

A 503 model_unavailable is the case where they did, and it is worth a backoff retry
before anything else.

What is not a rate limit

502 upstream_error is a provider failure, not throttling. Retrying is still right, but
a tight loop makes it worse.

504 request_timeout is our clock. The fix is a longer budget or a shorter prompt.

A burst of 400s across every account at once is your payload, not the pool. A
request-shaped failure is deterministic, so it fails identically everywhere; the gateway
stops at the first one rather than multiplying it.

Previous Model capabilities