Streaming

Streaming

Add "stream": true to any of the three POST endpoints. The server answers
text/event-stream, one data: {...} frame per delta, terminated by data: [DONE].

Every OpenAI and Anthropic SDK handles this unchanged. This page is what the wire
actually looks like, so you can debug it with curl instead of guessing.

A complete exchange

curl -N https://hisoka.ai/api/v1/chat/completions \
  -H "Authorization: Bearer $HISOKA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "glm-5.3-flash",
    "stream": true,
    "messages": [{ "role": "user", "content": "Count to three." }]
  }'

data: {"id":"chatcmpl-01M...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":null}]}

data: {"id":"chatcmpl-01M...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"One"},"finish_reason":null}]}

data: {"id":"chatcmpl-01M...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":", two, three."},"finish_reason":null}]}

data: {"id":"chatcmpl-01M...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]

The last frame carries your bill

The final frame before [DONE] includes the same hisoka block a non-streamed
response carries, so a streamed session is reconcilable without a second request:

{
  "choices": [],
  "usage": { "prompt_tokens": 18, "completion_tokens": 9, "total_tokens": 27 },
  "hisoka": {
    "model": "glm-5.3-flash",
    "usd_charged": 0.000015,
    "usd_micro_charged": 15,
    "balance": "4.819971"
  }
}

Tokens are forwarded as they arrive

The first visible token reaches you in well under a second on a model that supports it.
Events are held only until the first visible text or tool call, then forwarded live, so a
client that measures time-to-first-token is measuring the model rather than the gateway.

A buffered stream is a bug worth reporting: it appears as a frozen UI that dumps the
whole answer at the end.

Reasoning models in a stream

A reasoning model may stream reasoning, then content, then finish. Two shapes are
legitimate and must not be treated as failures:

The second case still emits its terminator, so your client is never left holding an open
turn. Retrying it pays twice for an answer already delivered.

A stream with no content, no tool calls and no reasoning is the real failure.

Tool calls in a stream

A tool call arrives as frames that build up one call:

data: {"choices":[{"delta":{"tool_calls":[{"index":0,"id":"call-1","type":"function","function":{"name":"read_file","arguments":""}}]}}]}

data: {"choices":[{"delta":{"tool_calls":[{"index":0,"function":{"arguments":"{\"path\":"}}]}}]}

data: {"choices":[{"delta":{"tool_calls":[{"index":0,"function":{"arguments":"\"/etc/hostname\"}"}}]}}]}

The name appears once, on the frame that opens the call. Later frames carry
arguments only: no repeated name, no repeated id. A client that assigns rather than
concatenates therefore still has the name, which is the difference between an agent
working and an agent reporting that the model produced no valid output.

Arguments arrive as string fragments and must be concatenated before parsing.

Cancelling a stream

Close the connection. The upstream request is abandoned with it, and the call is billed
for what was actually generated up to that point, not for the full completion.

Debugging with curl

-N disables curl’s output buffering. Without it the frames arrive at once at the end and
a healthy stream looks frozen.

curl -N -s https://hisoka.ai/api/v1/chat/completions \
  -H "Authorization: Bearer $HISOKA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"glm-5.3-flash","stream":true,"messages":[{"role":"user","content":"hi"}]}' \
  | while IFS= read -r line; do printf '%s %s\n' "$(date +%H:%M:%S.%3N)" "$line"; done

Timestamps on each line show whether tokens arrive progressively or in one burst.