When switching from a Vision-enabled Agent (like GPT-40) to a non-Vision-enabled model (like O3 Mini), an error occurs:

“Sorry, OpenAI has rejected your request. Here is the error message from OpenAI: Invalid content type. image_url is only supported by certain models.”

Scenario:

I use GPT-40 with Vision Support to handle images and convert attachments to text successfully.

When switching back to O3 Mini, it tries to process the image and fails with the above error.

Expected Behavior:

Images should be excluded from being sent to models that do not support them. This would prevent errors and ensure smoother transitions between models.

Current Workaround:

Deleting all image-related messages before resuming with a non-Vision model.

Proposed Solution:

Introduce logic to exclude image data when interacting with models that don't support vision capabilities.