Failover and uptime

Failover and uptime

A model on this gateway is not one connection to one provider. It is a pool: several
accounts behind the same model id, and a request that fails on one is retried on another
inside the same call.

You do not configure this. It is what the pool is for, and knowing how it behaves tells you
when an error is worth reporting.

What a request goes through

  1. Pool selection. The model id is resolved to the accounts that can serve it.
  2. Attempt. The request goes to one account.
  3. Retry on a retryable failure. 429, a 5xx, or a transport failure moves to the
    next account.
  4. Escalating passes. If the whole pool is busy or cooling down, the gateway waits and
    tries again, roughly twelve seconds of tolerance in total.
  5. Give up. Only then do you see 429 pool_exhausted or 503 model_unavailable.

A request-shaped failure stops instead of retrying. A 400 will fail identically on the
next account, so retrying it would turn one malformed request into several failed attempts
and make a provider look broken.

What you see, and what it means

You see Meaning
200 normal, possibly after internal retries you never noticed
429 rate_limited one upstream account is throttled, wait carries Retry-After
429 pool_exhausted every account for that model is throttled at once
502 upstream_error the provider failed in a way retries did not fix
503 model_unavailable nothing healthy left to try, or an operator-side credential problem
504 request_timeout the request ran past its time limit
Most failures never reach you

A provider flap writes many rows in telemetry and few errors at the client, because the
retry ladder absorbs them. When the operator reports a spike, the number that matters is
how many failed after a retry, not how many attempts failed. That distinction is why “the
uptime dropped” and “I got errors” are different measurements.

Reading Retry-After

Every 429 carries it, and every failure carries x-should-retry. When Retry-After
says 6 seconds, the pool has an account free in 6 seconds. Waiting is nearly always better
than switching models, because switching costs you a model choice for a wait you could
have sat through.

Health is per account, not per model

An account that fails repeatedly is benched briefly so a burst of traffic does not keep
landing on a broken connection. Traffic then goes to its siblings. This is invisible to
you except as a 200 that took slightly longer.

An account that is genuinely dead stops receiving traffic until an operator fixes it. That
is when a model may vanish from GET /models rather than stay listed and fail: an
out-of-stock model is worse than an absent one, because a buyer who paid for it needs a
refund.

When to report a problem

Report with the request_id when:

  • you get the same code repeatedly for several minutes, not one blip
  • one model fails while another works, and both are in the catalogue
  • a 5xx repeats on a request that has worked before

Do not report a single 429 or a single 502: read Retry-After, retry, and only report
it if the retry also fails.

Measuring it yourself

ai_supplier_logs is operator telemetry, but your own view is the ledger: a failed call
costs nothing. If a batch of requests failed, GET /transactions will not show charges for
them, which is the fastest way to confirm you were not billed for an error.

curl -s "https://hisoka.ai/api/v1/transactions?limit=5&type=usage" \
  -H "Authorization: Bearer $HISOKA_API_KEY" | jq '.data[] | {model, amount_micro_usd, created_at}'

A gap where your failed requests would be is the confirmation that a failure at the gateway
is not a charge against your balance.

Previous Wallet Next Rate limits