{"title":"Exercise: Vision Models and Image Understanding","description":null,"content":"# Exercise 1: Extract Structured Data from an Image\n\nSend an image to a vision model and get back structured JSON, not a paragraph. Ask for specific fields and parse the result into a dictionary you can use downstream.\n\nVision models are most useful in applications when they return data, not prose. Instead of \"describe this image,\" you ask for the exact fields you need (subject, colors, setting) in JSON, and the model fills them in from what it sees. This turns an image into a record you can store, filter, or route, the same way structured output does for text.\n\n### Requirements\n\n- Send a publicly hosted image to a vision-capable model.\n- Instruct it to return a JSON object with `subject` (string), `primary_colors` (list of strings), and `setting` (string).\n- Request JSON with `response_format={\"type\": \"json_object\"}`.\n- Parse the response and print the fields.\n\n\n**main.py**\n\n```python\nfrom openai import OpenAI\nimport json\n\nclient = OpenAI() # Uses OPENAI_API_KEY from environment\n\nimage_url = \"https://upload.wikimedia.org/wikipedia/commons/1/15/Cat_August_2010-4.jpg\"\n\n# TODO: call a vision model with a text instruction and the image_url,\n# asking for JSON with subject, primary_colors, and setting.\n# Use response_format json_object.\n\n# TODO: parse the response with json.loads and print each field\n```\n\n\n### Expected Output\n\nThe model reads the image and fills in your schema.\n\n\n```plaintext\nSubject: a domestic cat\nPrimary colors: ['gray', 'white']\nSetting: outdoors on grass\n```\n\n\n### Solution\n\n\n**main.py**\n\n```python\nfrom openai import OpenAI\nimport json\n\nclient = OpenAI()\n\nimage_url = \"https://upload.wikimedia.org/wikipedia/commons/1/15/Cat_August_2010-4.jpg\"\n\nresponse = client.chat.completions.create(\n model=\"gpt-5.4-mini\",\n messages=[\n {\n \"role\": \"user\",\n \"content\": [\n {\n \"type\": \"text\",\n \"text\": (\n \"Return a JSON object describing this image with keys: \"\n \"subject (string), primary_colors (list of strings), setting (string).\"\n ),\n },\n {\"type\": \"image_url\", \"image_url\": {\"url\": image_url}},\n ],\n }\n ],\n response_format={\"type\": \"json_object\"},\n)\n\ndata = json.loads(response.choices[0].message.content)\nprint(f\"Subject: {data['subject']}\")\nprint(f\"Primary colors: {data['primary_colors']}\")\nprint(f\"Setting: {data['setting']}\")\n```\n\n\nThe image is passed as an `image_url` content part alongside the text instruction, which is how a chat model receives pixels. Asking for named fields plus `response_format` json turns the model's visual understanding into a typed record rather than a description you would have to parse yourself. The fields you choose are the contract: the model fills exactly what you ask for, so designing that schema is the real work in a vision feature.\n\n---\n\n# Exercise 2: Compare Two Images in One Call\n\nA vision model can reason over several images at once. Send two images in a single message and ask the model to compare them, here, to decide which one shows food. Multi-image prompts are how you build features like \"which of these matches?\" or \"what changed between these two frames?\".\n\n### Requirements\n\n- Put two image URLs in one user message, each as an `image_url` content part.\n- Ask a question that requires comparing them.\n- Print the model's answer.\n\n\n**main.py**\n\n```python\nfrom openai import OpenAI\nimport json\n\nclient = OpenAI() # Uses OPENAI_API_KEY from environment\n\nimage_a = \"https://upload.wikimedia.org/wikipedia/commons/1/15/Cat_August_2010-4.jpg\"\nimage_b = \"https://upload.wikimedia.org/wikipedia/commons/d/d3/Supreme_pizza.jpg\"\n\n# TODO: send both images in one message and ask which one shows food.\n# Print the model's answer.\n```\n\n\n### Expected Output\n\nThe model identifies the pizza image as the one showing food and explains briefly.\n\n\n```plaintext\nImage B shows food: it is a supreme pizza with various toppings. Image A is a cat, not food.\n```\n\n\n### Solution\n\n\n**main.py**\n\n```python\nfrom openai import OpenAI\nimport json\n\nclient = OpenAI() # Uses OPENAI_API_KEY from environment\n\nimage_a = \"https://upload.wikimedia.org/wikipedia/commons/1/15/Cat_August_2010-4.jpg\"\nimage_b = \"https://upload.wikimedia.org/wikipedia/commons/d/d3/Supreme_pizza.jpg\"\n\nresponse = client.chat.completions.create(\n model=\"gpt-5.4-mini\",\n messages=[{\"role\": \"user\", \"content\": [\n {\"type\": \"text\", \"text\": \"Image A and Image B follow. Which one shows food? Answer A or B and say why.\"},\n {\"type\": \"image_url\", \"image_url\": {\"url\": image_a}},\n {\"type\": \"image_url\", \"image_url\": {\"url\": image_b}},\n ]}],\n)\nprint(response.choices[0].message.content)\n```\n\n\nMultiple `image_url` content parts in one message let the model see and compare all of them together, which a single-image call cannot do. Labeling them in the text (\"Image A and Image B\") gives the model a way to refer to each in its answer. This multi-image pattern underlies visual comparison, before-and-after analysis, and choosing the best match from a set. It needs a vision-capable model and network access to the images.","pageType":"ai-engineering"}

Get Premium