Exercise: Vision Models and Image Understanding
Exercise 1: Extract Structured Data from an Image
Send an image to a vision model and get back structured JSON, not a paragraph. Ask for specific fields and parse the result into a dictionary you can use downstream.
Vision models are most useful in applications when they return data, not prose. Instead of "describe this image," you ask for the exact fields you need (subject, colors, setting) in JSON, and the model fills them in from what it sees. This turns an image into a record you can store, filter, or route, the same way structured output does for text.
Requirements
- Send a publicly hosted image to a vision-capable model.
- Instruct it to return a JSON object with
subject(string),primary_colors(list of strings), andsetting(string). - Request JSON with
response_format={"type": "json_object"}. - Parse the response and print the fields.
Expected Output
The model reads the image and fills in your schema.
Solution
The image is passed as an image_url content part alongside the text instruction, which is how a chat model receives pixels. Asking for named fields plus response_format json turns the model's visual understanding into a typed record rather than a description you would have to parse yourself. The fields you choose are the contract: the model fills exactly what you ask for, so designing that schema is the real work in a vision feature.
Exercise 2: Compare Two Images in One Call
A vision model can reason over several images at once. Send two images in a single message and ask the model to compare them, here, to decide which one shows food. Multi-image prompts are how you build features like "which of these matches?" or "what changed between these two frames?".
Requirements
- Put two image URLs in one user message, each as an
image_urlcontent part. - Ask a question that requires comparing them.
- Print the model's answer.
Expected Output
The model identifies the pizza image as the one showing food and explains briefly.
Solution
Multiple image_url content parts in one message let the model see and compare all of them together, which a single-image call cannot do. Labeling them in the text ("Image A and Image B") gives the model a way to refer to each in its answer. This multi-image pattern underlies visual comparison, before-and-after analysis, and choosing the best match from a set. It needs a vision-capable model and network access to the images.