{"title":"Exercise: Video Understanding and Generation","description":null,"content":"# Exercise 1: Summarize a Video from Sampled Frames\n\nUnderstand a video by sampling a few frames, describing each with a vision model, and summarizing the descriptions into an account of what happens across the clip.\n\nModels do not watch video directly; the standard approach is to sample frames at intervals and treat them as a sequence of images. Each frame gets a description, and a language model stitches those descriptions into a narrative. Sampling is the key decision: too few frames and you miss the action, too many and you waste tokens on near-identical images.\n\n### Requirements\n\n- Use a small set of frame image URLs that represent samples from a video, in order.\n- Describe each frame with a vision model.\n- Pass the ordered descriptions to a chat model and ask for a summary of what happens across the video.\n- Print each frame description and the final summary.\n\n\n**main.py**\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI() # Uses OPENAI_API_KEY from environment\n\n# Frames sampled from a video, in chronological order (URLs stand in for extracted frames)\nframe_urls = [\n \"https://upload.wikimedia.org/wikipedia/commons/1/15/Cat_August_2010-4.jpg\",\n \"https://upload.wikimedia.org/wikipedia/commons/d/d3/Supreme_pizza.jpg\",\n]\n\ndef describe_frame(url: str) -> str:\n # TODO: call a vision model to describe this frame in one short sentence\n pass\n\n# TODO: describe each frame in order\n\n# TODO: pass the ordered descriptions to a chat model and ask for a summary\n# of what happens across the video, then print the descriptions and summary\n```\n\n\n### Expected Output\n\nEach frame is described, and the summary reads the descriptions as a sequence.\n\n\n```plaintext\nFrame 1: A gray and white cat sitting outdoors.\nFrame 2: A supreme pizza with various toppings.\nSummary: The clip opens on a cat outdoors and then cuts to a close-up of a supreme pizza.\n```\n\n\n### Solution\n\n\n**main.py**\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI()\n\nframe_urls = [\n \"https://upload.wikimedia.org/wikipedia/commons/1/15/Cat_August_2010-4.jpg\",\n \"https://upload.wikimedia.org/wikipedia/commons/d/d3/Supreme_pizza.jpg\",\n]\n\ndef describe_frame(url: str) -> str:\n response = client.chat.completions.create(\n model=\"gpt-5.4-mini\",\n messages=[\n {\n \"role\": \"user\",\n \"content\": [\n {\"type\": \"text\", \"text\": \"Describe this frame in one short sentence.\"},\n {\"type\": \"image_url\", \"image_url\": {\"url\": url}},\n ],\n }\n ],\n )\n return response.choices[0].message.content.strip()\n\ndescriptions = [describe_frame(url) for url in frame_urls]\nfor i, description in enumerate(descriptions, start=1):\n print(f\"Frame {i}: {description}\")\n\nnumbered = \"\\n\".join(f\"Frame {i}: {d}\" for i, d in enumerate(descriptions, start=1))\nsummary_response = client.chat.completions.create(\n model=\"gpt-5.4-mini\",\n messages=[\n {\"role\": \"system\", \"content\": \"Summarize what happens across these ordered video frames.\"},\n {\"role\": \"user\", \"content\": numbered},\n ],\n)\nprint(f\"Summary: {summary_response.choices[0].message.content.strip()}\")\n```\n\n\nSampling reduces a video to a manageable sequence of images, and keeping the frames in order is what lets the summarizer describe change over time rather than a bag of unrelated pictures. The per-frame descriptions are the bridge: once the visual content is text, the summary step is ordinary text work. The number of frames you sample is the lever that trades cost against how much of the action you capture.\n\n---\n\n# Exercise 2: Answer a Question About a Video\n\nOnce a video is captioned frame by frame, you can do more than summarize it: you can answer questions about it. Caption the sampled frames, then answer a specific question using those captions as context. This is video question-answering built on the same caption-then-reason pipeline.\n\n### Requirements\n\n- Caption each frame in order with a vision model.\n- Combine the ordered captions as context.\n- Answer a specific question about the video using only the captions, and print the answer.\n\n\n**main.py**\n\n```python\nfrom openai import OpenAI\nimport json\n\nclient = OpenAI() # Uses OPENAI_API_KEY from environment\n\nframe_urls = [\n \"https://upload.wikimedia.org/wikipedia/commons/1/15/Cat_August_2010-4.jpg\",\n \"https://upload.wikimedia.org/wikipedia/commons/d/d3/Supreme_pizza.jpg\",\n]\nquestion = \"Does a cat appear in this video?\"\n\ndef describe_frame(url):\n # TODO: caption this frame in one short sentence\n ...\n\n# TODO: caption each frame in order, then answer the question using the ordered captions as context\n```\n\n\n### Expected Output\n\nThe captions establish that a cat appears in the first frame, and the answer reflects that.\n\n\n```plaintext\nYes. The first frame shows a cat sitting outdoors, so a cat does appear in the video.\n```\n\n\n### Solution\n\n\n**main.py**\n\n```python\nfrom openai import OpenAI\nimport json\n\nclient = OpenAI() # Uses OPENAI_API_KEY from environment\n\nframe_urls = [\n \"https://upload.wikimedia.org/wikipedia/commons/1/15/Cat_August_2010-4.jpg\",\n \"https://upload.wikimedia.org/wikipedia/commons/d/d3/Supreme_pizza.jpg\",\n]\nquestion = \"Does a cat appear in this video?\"\n\ndef describe_frame(url):\n response = client.chat.completions.create(\n model=\"gpt-5.4-mini\",\n messages=[{\"role\": \"user\", \"content\": [\n {\"type\": \"text\", \"text\": \"Describe this frame in one short sentence.\"},\n {\"type\": \"image_url\", \"image_url\": {\"url\": url}}]}],\n )\n return response.choices[0].message.content.strip()\n\ncaptions = [f\"Frame {i+1}: {describe_frame(u)}\" for i, u in enumerate(frame_urls)]\nanswer = client.chat.completions.create(\n model=\"gpt-5.4-mini\",\n messages=[{\"role\": \"system\", \"content\": \"Answer the question using only these ordered video frame descriptions.\"},\n {\"role\": \"user\", \"content\": chr(10).join(captions) + f\"\\n\\nQuestion: {question}\"}],\n)\nprint(answer.choices[0].message.content.strip())\n```\n\n\nQuestion-answering reuses the frame captions as evidence, the same way RAG answers from retrieved chunks: the model reasons over the text descriptions rather than the pixels. Keeping the frames ordered lets it answer questions about sequence and change, not just presence. The captioning prompt sets the ceiling on what you can ask, since a detail no caption mentions is invisible to the answer step. It needs a vision-capable model and access to the frame URLs.","pageType":"ai-engineering"}

Get Premium