Make a set of images searchable by text. Use a vision model to describe each image, embed those descriptions into the same vector space as a text query, and retrieve the image whose description best matches the query.

This is the most common way to bring images into a RAG pipeline. Rather than building a separate image search system, you turn each image into a text description with a vision model, then reuse the text embedding and retrieval machinery you already have. The retrieval quality depends entirely on how well the descriptions capture what a user might search for.

Requirements

Expected Output

Each image gets a description, and the landmark query retrieves the tower image even though the query never mentions a tower.

Solution

The vision model converts each image into a sentence that lives in the same space as text, so a single embedding model can compare the query against all of them. Everything after the describe step is ordinary text retrieval, which is exactly the appeal: one pipeline handles both modalities, and the only multimodal-specific piece is the captioning call. The cost is that anything the description omits becomes invisible to search, so the prompt you give the vision model directly shapes what users can find.

Extend image retrieval into a full multimodal RAG answer: caption the images, retrieve the one most relevant to a question, then answer the question using that image's description as context. The model never sees the pixels at answer time, only the caption that retrieval selected.

Requirements

Expected Output

The landmark question retrieves the Eiffel Tower image, and the answer comes from its caption.

Solution

This is the complete multimodal RAG path: a vision model turns images into text, retrieval picks the relevant one, and a text model answers from it. Separating captioning from answering means the expensive vision call happens once at index time, while answering reuses ordinary text retrieval and generation. The answer is only as good as the caption, which is why the captioning prompt (what details to include) shapes what the system can later answer. This exercise needs a vision-capable model and network access to the images.