fix: support image input in OpenAI Chat user messages#26825
Conversation
Convert MediaPart to OpenAI image_url content block format in user messages. Adds schema for text and image_url content blocks. Updates lowerUserMessage to handle mixed text+media content. Adds tests for media handling. Fixes: Custom OpenAI-compatible providers image file attachments not reaching vision-capable models.
github-actions
Bot
added
the
needs:compliance
|
This PR doesn't fully meet our contributing guidelines and PR template. What needs to be fixed:
Please edit this PR description to address the above within 2 hours, or it will be automatically closed. If you believe this was flagged incorrectly, please let a maintainer know. |
|
Thanks for your contribution! This PR doesn't have a linked issue. All PRs must reference an existing issue. Please:
See CONTRIBUTING.md for details. |
|
The following comment was made by an LLM, it may be inaccurate: Potential Related PRs Found:
These PRs may be addressing overlapping functionality related to image support in OpenAI providers. PR #21627 is particularly relevant as it specifically mentions enabling image support for custom OpenAI-compatible providers, which aligns with the current PR's fix for "Custom OpenAI-compatible providers image file attachments." |
There was a problem hiding this comment.
Pull request overview
This PR updates the OpenAI Chat (/chat/completions) protocol lowering so LLM.user(...) messages containing image media are converted into OpenAI’s content: [{ type: "image_url", ... }] format, enabling vision-capable OpenAI-compatible providers to receive image attachments.
Changes:
- Extend the OpenAI Chat request schema to allow user
contentas either a string or an array of typed content blocks (text/image_url). - Update
lowerUserMessageto emit content blocks when media is present, convertingMediaPartinto adata:<mime>;base64,...URL. - Replace/expand tests to assert correct lowering for media-only, mixed text+media, and text-only user messages.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| packages/llm/src/protocols/openai-chat.ts | Adds OpenAI Chat content-block schema + lowers user media parts into image_url blocks. |
| packages/llm/test/provider/openai-chat.test.ts | Adds assertions for prepared OpenAI Chat bodies when user messages contain media and mixed content. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| type: "image_url" as const, | ||
| image_url: { | ||
| url: `data:${part.mediaType};base64,${mediaBytes(part)}`, | ||
| }, | ||
| } |
| if (hasMedia) { | ||
| const contentBlocks = message.content.map((part) => lowerUserPart(part as TextPart | MediaPart)) | ||
| return { role: "user" as const, content: contentBlocks } |
| it.effect("prepares user message with media as image_url content block", () => | ||
| Effect.gen(function* () { | ||
| const error = yield* LLMClient.prepare( | ||
| const prepared = yield* LLMClient.prepare<OpenAIChat.OpenAIChatBody>( | ||
| LLM.request({ | ||
| id: "req_media", | ||
| model, | ||
| messages: [LLM.user({ type: "media", mediaType: "image/png", data: "AAECAw==" })], | ||
| }), | ||
| ).pipe(Effect.flip) | ||
| ) |
Convert MediaPart to OpenAI image_url content block format in user messages.
Adds schema for text and image_url content blocks. Updates lowerUserMessage
to handle mixed text+media content. Adds tests for media handling.
Fixes: Custom OpenAI-compatible providers image file attachments not reaching
vision-capable models.