Understand a video by sampling a few frames, describing each with a vision model, and summarizing the descriptions into an account of what happens across the clip.

Models do not watch video directly; the standard approach is to sample frames at intervals and treat them as a sequence of images. Each frame gets a description, and a language model stitches those descriptions into a narrative. Sampling is the key decision: too few frames and you miss the action, too many and you waste tokens on near-identical images.

Requirements

Expected Output

Each frame is described, and the summary reads the descriptions as a sequence.

Solution

Sampling reduces a video to a manageable sequence of images, and keeping the frames in order is what lets the summarizer describe change over time rather than a bag of unrelated pictures. The per-frame descriptions are the bridge: once the visual content is text, the summary step is ordinary text work. The number of frames you sample is the lever that trades cost against how much of the action you capture.

Once a video is captioned frame by frame, you can do more than summarize it: you can answer questions about it. Caption the sampled frames, then answer a specific question using those captions as context. This is video question-answering built on the same caption-then-reason pipeline.

Requirements

Expected Output

The captions establish that a cat appears in the first frame, and the answer reflects that.

Solution

Question-answering reuses the frame captions as evidence, the same way RAG answers from retrieved chunks: the model reasons over the text descriptions rather than the pixels. Keeping the frames ordered lets it answer questions about sequence and change, not just presence. The captioning prompt sets the ceiling on what you can ask, since a detail no caption mentions is invisible to the answer step. It needs a vision-capable model and access to the frame URLs.