{"title":"Exercise: Multimodal RAG","description":null,"content":"# Exercise 1: Caption and Retrieve over Images\n\nMake a set of images searchable by text. Use a vision model to describe each image, embed those descriptions into the same vector space as a text query, and retrieve the image whose description best matches the query.\n\nThis is the most common way to bring images into a RAG pipeline. Rather than building a separate image search system, you turn each image into a text description with a vision model, then reuse the text embedding and retrieval machinery you already have. The retrieval quality depends entirely on how well the descriptions capture what a user might search for.\n\n### Requirements\n\n- For each image URL, ask a vision model to write a short factual description.\n- Embed every description with `text-embedding-3-small`.\n- Embed the text query and retrieve the image whose description is most similar.\n- Print each description and the URL of the best-matching image.\n\n\n**main.py**\n\n```python\nfrom openai import OpenAI\nimport math\n\nclient = OpenAI() # Uses OPENAI_API_KEY from environment\n\n# Public image URLs. If any URL fails to load, swap in another public image.\nimage_urls = [\n \"https://upload.wikimedia.org/wikipedia/commons/1/15/Cat_August_2010-4.jpg\",\n \"https://upload.wikimedia.org/wikipedia/commons/a/a8/Tour_Eiffel_Wikimedia_Commons.jpg\",\n \"https://upload.wikimedia.org/wikipedia/commons/d/d3/Supreme_pizza.jpg\",\n]\n\nquery = \"a famous landmark in Paris\"\n\ndef cosine_similarity(a, b):\n dot = sum(x * y for x, y in zip(a, b))\n return dot / (math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b)))\n\n# TODO: for each image URL, call the vision model to get a short description\n\n# TODO: embed the descriptions and the query with text-embedding-3-small\n\n# TODO: find the image whose description is most similar to the query\n\n# TODO: print each description and the URL of the best match\n```\n\n\n### Expected Output\n\nEach image gets a description, and the landmark query retrieves the tower image even though the query never mentions a tower.\n\n\n```plaintext\n[0] A close-up photo of a domestic cat sitting outdoors.\n[1] The Eiffel Tower, a tall iron lattice tower in Paris, against a blue sky.\n[2] A supreme pizza topped with pepperoni, peppers, and onions.\n\nBest match for \"a famous landmark in Paris\":\n https://upload.wikimedia.org/wikipedia/commons/a/a8/Tour_Eiffel_Wikimedia_Commons.jpg\n```\n\n\n### Solution\n\n\n**main.py**\n\n```python\nfrom openai import OpenAI\nimport math\n\nclient = OpenAI()\n\nimage_urls = [\n \"https://upload.wikimedia.org/wikipedia/commons/1/15/Cat_August_2010-4.jpg\",\n \"https://upload.wikimedia.org/wikipedia/commons/a/a8/Tour_Eiffel_Wikimedia_Commons.jpg\",\n \"https://upload.wikimedia.org/wikipedia/commons/d/d3/Supreme_pizza.jpg\",\n]\n\nquery = \"a famous landmark in Paris\"\n\ndef cosine_similarity(a, b):\n dot = sum(x * y for x, y in zip(a, b))\n return dot / (math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b)))\n\ndef describe(url: str) -> str:\n response = client.chat.completions.create(\n model=\"gpt-5.5\",\n messages=[\n {\n \"role\": \"user\",\n \"content\": [\n {\"type\": \"text\", \"text\": \"Describe this image in one short factual sentence.\"},\n {\"type\": \"image_url\", \"image_url\": {\"url\": url}},\n ],\n }\n ],\n )\n return response.choices[0].message.content.strip()\n\ndescriptions = [describe(url) for url in image_urls]\nfor i, description in enumerate(descriptions):\n print(f\"[{i}] {description}\")\n\ntexts = descriptions + [query]\nvectors = [\n item.embedding\n for item in client.embeddings.create(model=\"text-embedding-3-small\", input=texts).data\n]\ndescription_vectors, query_vector = vectors[:-1], vectors[-1]\n\nbest = max(\n range(len(image_urls)),\n key=lambda i: cosine_similarity(query_vector, description_vectors[i]),\n)\n\nprint(f'\\nBest match for \"{query}\":')\nprint(f\" {image_urls[best]}\")\n```\n\n\nThe vision model converts each image into a sentence that lives in the same space as text, so a single embedding model can compare the query against all of them. Everything after the `describe` step is ordinary text retrieval, which is exactly the appeal: one pipeline handles both modalities, and the only multimodal-specific piece is the captioning call. The cost is that anything the description omits becomes invisible to search, so the prompt you give the vision model directly shapes what users can find.\n\n---\n\n# Exercise 2: Caption, Retrieve, then Answer\n\nExtend image retrieval into a full multimodal RAG answer: caption the images, retrieve the one most relevant to a question, then answer the question using that image's description as context. The model never sees the pixels at answer time, only the caption that retrieval selected.\n\n### Requirements\n\n- Caption each image with a vision model.\n- Embed the captions and the question, and retrieve the most relevant caption.\n- Answer the question using only that caption as context, and print the chosen image and answer.\n\n\n**main.py**\n\n```python\nimport math\nfrom openai import OpenAI\n\nclient = OpenAI() # Uses OPENAI_API_KEY from environment\n\nimage_urls = [\n \"https://upload.wikimedia.org/wikipedia/commons/1/15/Cat_August_2010-4.jpg\",\n \"https://upload.wikimedia.org/wikipedia/commons/a/a8/Tour_Eiffel_Wikimedia_Commons.jpg\",\n]\n\nquestion = \"In which city is the landmark located?\"\n\ndef cosine_similarity(a, b):\n dot = sum(x * y for x, y in zip(a, b))\n return dot / (math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b)))\n\ndef describe(url):\n # TODO: caption the image with a vision model in one sentence\n ...\n\n# TODO: caption both images, embed the captions, retrieve the caption most relevant to the question,\n# then answer the question using that caption as context\n```\n\n\n### Expected Output\n\nThe landmark question retrieves the Eiffel Tower image, and the answer comes from its caption.\n\n\n```plaintext\nBest image: https://upload.wikimedia.org/wikipedia/commons/a/a8/Tour_Eiffel_Wikimedia_Commons.jpg\nAnswer: The landmark is the Eiffel Tower, which is located in Paris.\n```\n\n\n### Solution\n\n\n**main.py**\n\n```python\nimport math\nfrom openai import OpenAI\n\nclient = OpenAI() # Uses OPENAI_API_KEY from environment\n\nimage_urls = [\n \"https://upload.wikimedia.org/wikipedia/commons/1/15/Cat_August_2010-4.jpg\",\n \"https://upload.wikimedia.org/wikipedia/commons/a/a8/Tour_Eiffel_Wikimedia_Commons.jpg\",\n]\n\nquestion = \"In which city is the landmark located?\"\n\ndef cosine_similarity(a, b):\n dot = sum(x * y for x, y in zip(a, b))\n return dot / (math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b)))\n\ndef describe(url):\n response = client.chat.completions.create(\n model=\"gpt-5.5\",\n messages=[{\"role\": \"user\", \"content\": [\n {\"type\": \"text\", \"text\": \"Describe this image in one sentence.\"},\n {\"type\": \"image_url\", \"image_url\": {\"url\": url}}]}],\n )\n return response.choices[0].message.content.strip()\n\ncaptions = [describe(url) for url in image_urls]\nvectors = [item.embedding for item in client.embeddings.create(model=\"text-embedding-3-small\", input=captions + [question]).data]\ncaption_vecs, qv = vectors[:-1], vectors[-1]\nbest = max(range(len(captions)), key=lambda i: cosine_similarity(qv, caption_vecs[i]))\n\nanswer = client.chat.completions.create(\n model=\"gpt-5.4-mini\",\n messages=[{\"role\": \"system\", \"content\": \"Answer the question using only the image description.\"},\n {\"role\": \"user\", \"content\": f\"Image: {captions[best]}\\n\\nQuestion: {question}\"}],\n)\nprint(\"Best image:\", image_urls[best])\nprint(\"Answer:\", answer.choices[0].message.content.strip())\n```\n\n\nThis is the complete multimodal RAG path: a vision model turns images into text, retrieval picks the relevant one, and a text model answers from it. Separating captioning from answering means the expensive vision call happens once at index time, while answering reuses ordinary text retrieval and generation. The answer is only as good as the caption, which is why the captioning prompt (what details to include) shapes what the system can later answer. This exercise needs a vision-capable model and network access to the images.","pageType":"ai-engineering"}

Get Premium