Vision & Documents

Send images and documents alongside text for multimodal understanding and analysis.

Overview

Multimodal-capable models can analyze images and documents alongside text. With the OpenAI-compatible Chat Completions API you supply these as an array of content parts in a message's content; with the Anthropic Messages API you use image and document content blocks. Yunxin routes each part to the underlying provider's native format automatically.

Vision and document support is model-dependent. Query GET /v1/models and check each model's capabilities array (e.g. vision) before sending non-text parts.

Content parts (Chat Completions)

The content field accepts an array of parts. Three input part types are supported:

Type	Description	Source
`text`	Plain text	inline string
`image_url`	Image for visual understanding	HTTPS URL or `data:` base64 URI
`file_url`	Document (e.g. PDF) for analysis	HTTPS URL or `data:` base64 URI

There is no generic video_url input part. Vision means image understanding via image_url; documents are sent via file_url. Video understanding is provider/model-specific and not exposed as an input content part, and video generation is a separate endpoint.

Sending images

Include an image_url part in the message content:

{
  "model": "model-id",
  "messages": [
    {
      "role": "user",
      "content": [
        {"type": "text", "text": "What's in this image?"},
        {"type": "image_url", "image_url": {"url": "https://example.com/photo.jpg"}}
      ]
    }
  ]
}

Base64 images

When you don't have a public URL, inline the image as a data: URI:

import os, base64
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["YUNXIN_API_KEY"],
    base_url="https://api.yuhuanstudio.com/v1",
)

with open("image.png", "rb") as f:
    b64 = base64.b64encode(f.read()).decode()

response = client.chat.completions.create(
    model="model-id",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Describe this image."},
            {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64}"}},
        ],
    }],
)

print(response.choices[0].message.content)

Multiple images

Send several images in one message:

response = client.chat.completions.create(
    model="model-id",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Compare these two images."},
            {"type": "image_url", "image_url": {"url": "https://example.com/image1.jpg"}},
            {"type": "image_url", "image_url": {"url": "https://example.com/image2.jpg"}},
        ],
    }],
)

Image detail

Some OpenAI-family models accept a detail hint controlling how much resolution is analyzed. It is optional and only honored by models that support it:

{
  "type": "image_url",
  "image_url": {
    "url": "https://example.com/photo.jpg",
    "detail": "high"
  }
}

Detail	Behavior
`low`	Faster, fewer tokens, reduced resolution.
`high`	Full-resolution analysis, more tokens.
`auto`	The model decides (default).

Document analysis

Send documents (such as PDFs) with the file_url part:

{
  "model": "model-id",
  "messages": [
    {
      "role": "user",
      "content": [
        {"type": "text", "text": "Summarize the key findings in this document."},
        {"type": "file_url", "file_url": {"url": "https://example.com/report.pdf"}}
      ]
    }
  ]
}

Files from the File API

Upload large files once via the File API and reference them by their content URL. The gateway resolves it automatically (presigned URL, otherwise base64):

file = client.files.create(file=open("report.pdf", "rb"), purpose="document")

response = client.chat.completions.create(
    model="model-id",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Summarize this report."},
            {"type": "file_url", "file_url": {
                "url": f"https://api.yuhuanstudio.com/v1/files/{file.id}/content"
            }},
        ],
    }],
)

Anthropic Messages format

The Messages API expresses the same inputs as typed content blocks. Images use an image block (base64 or URL source); documents use a document block:

import os
from anthropic import Anthropic

client = Anthropic(
    api_key=os.environ["YUNXIN_API_KEY"],
    # The Anthropic SDK appends /v1/messages itself — host only, no /v1.
    base_url="https://api.yuhuanstudio.com",
)

import base64

with open("image.png", "rb") as f:
    b64 = base64.b64encode(f.read()).decode()

message = client.messages.create(
    model="model-id",
    max_tokens=1024,
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "What's in this image?"},
            {"type": "image", "source": {
                "type": "base64", "media_type": "image/png", "data": b64,
            }},
        ],
    }],
)

print(message.content[0].text)

Supported image and document formats are determined by the upstream provider. Use the vision capability as a general indicator of multimodal support, and confirm specifics for the model you target via GET /v1/models/{model_id}.

Overview

Vision and document support is model-dependent. Query GET /v1/models and check each model's capabilities array (e.g. vision) before sending non-text parts.

Content parts (Chat Completions)

The content field accepts an array of parts. Three input part types are supported:

Type	Description	Source
`text`	Plain text	inline string
`image_url`	Image for visual understanding	HTTPS URL or `data:` base64 URI
`file_url`	Document (e.g. PDF) for analysis	HTTPS URL or `data:` base64 URI

Sending images

Include an image_url part in the message content:

{
  "model": "model-id",
  "messages": [
    {
      "role": "user",
      "content": [
        {"type": "text", "text": "What's in this image?"},
        {"type": "image_url", "image_url": {"url": "https://example.com/photo.jpg"}}
      ]
    }
  ]
}

Base64 images

When you don't have a public URL, inline the image as a data: URI:

import os, base64
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["YUNXIN_API_KEY"],
    base_url="https://api.yuhuanstudio.com/v1",
)

with open("image.png", "rb") as f:
    b64 = base64.b64encode(f.read()).decode()

response = client.chat.completions.create(
    model="model-id",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Describe this image."},
            {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64}"}},
        ],
    }],
)

print(response.choices[0].message.content)

Multiple images

Send several images in one message:

response = client.chat.completions.create(
    model="model-id",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Compare these two images."},
            {"type": "image_url", "image_url": {"url": "https://example.com/image1.jpg"}},
            {"type": "image_url", "image_url": {"url": "https://example.com/image2.jpg"}},
        ],
    }],
)

Image detail

Some OpenAI-family models accept a detail hint controlling how much resolution is analyzed. It is optional and only honored by models that support it:

{
  "type": "image_url",
  "image_url": {
    "url": "https://example.com/photo.jpg",
    "detail": "high"
  }
}

Detail	Behavior
`low`	Faster, fewer tokens, reduced resolution.
`high`	Full-resolution analysis, more tokens.
`auto`	The model decides (default).

Document analysis

Send documents (such as PDFs) with the file_url part:

{
  "model": "model-id",
  "messages": [
    {
      "role": "user",
      "content": [
        {"type": "text", "text": "Summarize the key findings in this document."},
        {"type": "file_url", "file_url": {"url": "https://example.com/report.pdf"}}
      ]
    }
  ]
}

Files from the File API

Upload large files once via the File API and reference them by their content URL. The gateway resolves it automatically (presigned URL, otherwise base64):

file = client.files.create(file=open("report.pdf", "rb"), purpose="document")

response = client.chat.completions.create(
    model="model-id",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Summarize this report."},
            {"type": "file_url", "file_url": {
                "url": f"https://api.yuhuanstudio.com/v1/files/{file.id}/content"
            }},
        ],
    }],
)

Anthropic Messages format

The Messages API expresses the same inputs as typed content blocks. Images use an image block (base64 or URL source); documents use a document block:

import os
from anthropic import Anthropic

client = Anthropic(
    api_key=os.environ["YUNXIN_API_KEY"],
    # The Anthropic SDK appends /v1/messages itself — host only, no /v1.
    base_url="https://api.yuhuanstudio.com",
)

import base64

with open("image.png", "rb") as f:
    b64 = base64.b64encode(f.read()).decode()

message = client.messages.create(
    model="model-id",
    max_tokens=1024,
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "What's in this image?"},
            {"type": "image", "source": {
                "type": "base64", "media_type": "image/png", "data": b64,
            }},
        ],
    }],
)

print(message.content[0].text)

Overview

Content parts (Chat Completions)

Sending images

Base64 images

Multiple images

Image detail

Document analysis

Files from the File API

Anthropic Messages format

On this page

Vision & Documents

Overview

Content parts (Chat Completions)

Sending images

Base64 images

Multiple images

Image detail

Document analysis

Files from the File API

Anthropic Messages format

On this page