Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

Support image generation and output with Google and OpenAI #2970

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 42 commits into from
Oct 3, 2025
Merged

Support image generation and output with Google and OpenAI #2970

merged 42 commits into from
Oct 3, 2025

Conversation

DouweM
Copy link
Collaborator

@DouweM DouweM commented Sep 19, 2025 •
edited
Loading

Google

from pydantic_ai import Agent
from pydantic_ai.messages import BinaryImage

agent = Agent('google-gla:gemini-2.5-flash-image-preview', output_type=BinaryImage)

result = agent.run_sync('Generate an image of a programmer axolotl wearing Pydantic swag.')

output = result.output
assert isinstance(output, BinaryImage)
assert output.media_type == 'image/png'
assert isinstance(output.data, bytes)
assert result.response.images == [output]
with open('axolotl-google.png', 'wb') as f:
    f.write(output.data)
axolotl-google

OpenAI

from pydantic_ai import Agent
from pydantic_ai.messages import BinaryImage

agent = Agent('openai-responses:gpt-5', output_type=BinaryImage)

result = agent.run_sync('Generate an image of a programmer axolotl wearing Pydantic swag.')

output = result.output
assert isinstance(output, BinaryImage)
assert output.media_type == 'image/png'
assert isinstance(output.data, bytes)
assert result.response.images == [output]
with open('axolotl-openai.png', 'wb') as f:
    f.write(output.data)
axolotl-openai

To do

  • Image as result.output via output_type=Image
  • output_type=Image | str or other combinations
  • Streaming
  • OpenAI image generation tool options
  • Store OpenAI revised_prompt as built-in tool call result
  • Vertex AI
  • OTel
  • Temporal (doesn't work because of 2MB size limit)
  • Docs

@DouweM DouweM self-assigned this Sep 19, 2025
This was referenced Sep 19, 2025
Closed
Closed
@github-actions GitHub Actions
Copy link

github-actions bot commented Sep 19, 2025 •
edited
Loading

Docs Preview

commit: 91e2283
Preview URL: https://350a26a8-pydantic-ai-previews.pydantic.workers.dev

@DouweM DouweM changed the title Generate FileParts for images generated by Google and OpenAI Support image output by Google and OpenAI Sep 19, 2025
@DouweM DouweM mentioned this pull request Sep 20, 2025
Closed
2 tasks
@DouweM DouweM mentioned this pull request Sep 29, 2025
Closed
DouweM added 5 commits October 1, 2025 18:38
@DouweM
Merge branch 'main' into image-generation
cc44d4f 
# Conflicts:
#	pydantic_ai_slim/pydantic_ai/_output.py
#	pydantic_ai_slim/pydantic_ai/builtin_tools.py
#	pydantic_ai_slim/pydantic_ai/models/openai.py
#	tests/models/test_openai_responses.py
@DouweM
fix errors after merge conflicts
bea8fe8
@DouweM
Simplify handling of response without expected output
dd71619
@DouweM
Test interaction of image generation with output and tool calls
93a3492
@DouweM
streaming
86d8f1e
@DouweM DouweM changed the title Support image output by Google and OpenAI Support image generation and output with Google and OpenAI Oct 1, 2025
DouweM
DouweM commented Oct 2, 2025
View reviewed changes
pydantic_ai_slim/pydantic_ai/_agent_graph.py
_messages.ModelRequest(parts=output_parts, instructions=instructions)
)

async def _handle_text_response(
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was moved up

DouweM
DouweM commented Oct 2, 2025
View reviewed changes
pydantic_ai_slim/pydantic_ai/_output.py
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file is not worth reviewing, I just moved some logic around to get the allows_image boolean everywhere we need it

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🫡

DouweM
DouweM commented Oct 2, 2025
View reviewed changes
pydantic_ai_slim/pydantic_ai/builtin_tools.py Show resolved Hide resolved
DouweM
DouweM commented Oct 2, 2025
View reviewed changes
pydantic_ai_slim/pydantic_ai/models/openai.py Outdated Show resolved Hide resolved
dmontagu
dmontagu reviewed Oct 3, 2025
View reviewed changes
pydantic_ai_slim/pydantic_ai/models/anthropic.py Outdated Show resolved Hide resolved
dmontagu
dmontagu reviewed Oct 3, 2025
View reviewed changes
pydantic_ai_slim/pydantic_ai/models/cohere.py Outdated Show resolved Hide resolved
dmontagu
dmontagu reviewed Oct 3, 2025
View reviewed changes
pydantic_ai_slim/pydantic_ai/models/gemini.py Outdated
# This is currently never returned from gemini
pass
elif isinstance(item, FilePart): # pragma: no cover
# Images generated by models are not sent back to models that don't themselves generate images.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# Images generated by models are not sent back to models that don't themselves generate images.
# Files generated by models are not sent back to models that don't themselves generate files.

dmontagu
dmontagu reviewed Oct 3, 2025
View reviewed changes
pydantic_ai_slim/pydantic_ai/models/google.py Outdated Show resolved Hide resolved
dmontagu
dmontagu reviewed Oct 3, 2025
View reviewed changes
pydantic_ai_slim/pydantic_ai/models/groq.py Outdated Show resolved Hide resolved
dmontagu
dmontagu reviewed Oct 3, 2025
View reviewed changes
pydantic_ai_slim/pydantic_ai/models/huggingface.py Outdated Show resolved Hide resolved
dmontagu
dmontagu reviewed Oct 3, 2025
View reviewed changes
pydantic_ai_slim/pydantic_ai/models/mistral.py Outdated Show resolved Hide resolved
dmontagu
dmontagu reviewed Oct 3, 2025
View reviewed changes
pydantic_ai_slim/pydantic_ai/models/openai.py Outdated Show resolved Hide resolved
dmontagu
dmontagu reviewed Oct 3, 2025
View reviewed changes
pydantic_ai_slim/pydantic_ai/profiles/google.py Outdated Show resolved Hide resolved
dmontagu
dmontagu reviewed Oct 3, 2025
View reviewed changes
pydantic_ai_slim/pydantic_ai/__init__.py Show resolved Hide resolved
dmontagu
dmontagu reviewed Oct 3, 2025
View reviewed changes
pydantic_ai_slim/pydantic_ai/messages.py Outdated Show resolved Hide resolved
dmontagu
dmontagu reviewed Oct 3, 2025
View reviewed changes
tests/models/test_google.py Show resolved Hide resolved
dmontagu
dmontagu reviewed Oct 3, 2025
View reviewed changes
tests/models/test_openai_responses.py Show resolved Hide resolved
dmontagu
dmontagu reviewed Oct 3, 2025
View reviewed changes
tests/test_agent.py Show resolved Hide resolved
@DouweM DouweM mentioned this pull request Oct 3, 2025
Draft
@DouweM DouweM merged commit 3db21d4 into main Oct 3, 2025
30 checks passed
@DouweM DouweM deleted the image-generation branch October 3, 2025 21:25
lionpeloux pushed a commit to lionpeloux/pydantic-ai that referenced this pull request Oct 14, 2025
@DouweM @lionpeloux
Support image generation and output with Google and OpenAI (pydantic#…
2d0ba24 
…2970)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dmontagu dmontagu dmontagu left review comments

Assignees

@DouweM DouweM

Labels

None yet

Projects

None yet

Milestone

No milestone

4 participants

@DouweM @Kludex @alexmojaki @dmontagu
Morty Proxy This is a proxified and sanitized view of the page, visit original site.