Multimodal input

Multimodal input

Three wire protocols accept media, but the model decides what it can read. Check
capabilities.modalities before sending, because a model that does not declare a
modality refuses it with 400 modality_not_supported before the request reaches a
provider.

curl -s https://hisoka.ai/api/v1/models \
  | jq '.data[] | {id, modalities: .capabilities.modalities}'

Why refusing early matters

An unknown part forwarded to a provider came back as an HTTP 500, which reads as our
outage rather than your request. A modality the model does declare has a working path
through; one it does not is refused at the edge with a message naming the model’s real
capabilities.

Silent dropping is the failure this prevents: a clip the gateway discarded still produced
a normal 200, and the caller paid for a reply that never saw it.

Images

{
  "model": "glm-5.3-flash",
  "messages": [{
    "role": "user",
    "content": [
      { "type": "text", "text": "What is in this screenshot?" },
      { "type": "image_url", "image_url": { "url": "data:image/png;base64,iVBORw0..." } }
    ]
  }]
}

An https:// URL works as well as a data URL, where the provider supports fetching one.

Accepted spellings for the part type: image_url, image, input_image. The gateway
normalises all three, so a client using its own dialect does not produce a provider error.

Video

{
  "role": "user",
  "content": [
    { "type": "text", "text": "Describe the motion in this clip." },
    { "type": "video_url", "video_url": { "url": "https://example.com/clip.mp4" } }
  ]
}

Accepted spellings: video_url, video, input_video. Only send video to a model whose
capabilities.modalities includes video; that is a short list, and the catalogue is the
authority rather than a blog post.

Audio

Accepted spellings: input_audio, audio. The shape follows the provider’s own object,
so a client that already works against an OpenAI-compatible provider needs no change here.

Sizing a request

Images and video become prompt tokens, and prompt tokens are what you pay for. Two habits
keep the bill predictable:

• Resize before sending. A 4096x4096 screenshot costs far more than the same screen at
  1920x1080, and the model reads both.
• Send the frames you need, not the whole recording. A video contributes tokens per
  sampled frame.

A useful check before blaming a model for a bad answer: look at the usage.prompt_tokens
in the reply. A megabyte of base64 that produces double-digit prompt_tokens means the
media never arrived, and that is worth reporting with the request_id.

What is refused

The body limit is the gateway’s, not the model’s; shrinking the attachment is usually
faster than raising it.