Vision models let AI systems work with visual evidence: photographs, screenshots, diagrams, scanned documents, charts, and individual video frames. A strong vision-language model can describe a scene, read visible text, reason about layout, compare images, and answer targeted questions about what appears in the input.

In production, these capabilities show up in document processing, product search, insurance claims, UI automation, accessibility tooling, medical imaging support, industrial inspection, and multimodal assistants. The engineering work is choosing what visual evidence to send, at what resolution, with what prompt, and how to validate the answer.

What happens between the image you upload and the answer you get back affects latency, cost, preprocessing, and failure modes.

Traditional LLMs process a sequence of text tokens. Vision-language models (VLMs) add a visual encoder that turns an image into a sequence of visual tokens. The language model then attends over both the visual tokens and the text tokens in the prompt.

The flow has four stages:

Image detail becomes tokens. Higher resolution and more images increase latency, cost, and rate-limit pressure. Low-detail inputs are cheaper, while high-detail or original-detail inputs give the model more visual evidence to inspect. Image preprocessing is part of system design.

Most major model providers now support image input, but the API details differ. Some providers accept full video or document files; others expect image URLs, uploaded file IDs, or base64-encoded data. The core pattern is the same: combine text instructions with one or more visual inputs.

OpenAI Vision Models

OpenAI accepts images as URLs, uploaded files, or base64-encoded data depending on the API surface you use. The Responses API takes input as a list of messages, where each message can contain typed blocks such as input_text and input_image.

Text and image blocks sit in the same message, and the model receives them together. For new systems, check the provider's current Responses API or equivalent; newer multimodal features often appear there first.

For local images, you base64-encode them:

Base64 works when the image is already in your application process. For larger payloads, use uploaded file IDs or signed object-storage URLs when the provider supports them. They avoid large JSON payloads, reduce memory pressure, and make retries cleaner.

Vision models do not usually reason over raw pixels in the form you uploaded them. Providers resize, tile, patch, and tokenize images internally. That preprocessing affects the token count, which affects both cost and latency.

How OpenAI Handles Image Resolution

OpenAI exposes a detail parameter on image inputs for many vision-capable models:

The cost rules vary by model family. Current GPT-5.5 vision inputs use patch-based accounting, while older GPT-4o and GPT-4.1 models use tile-based accounting. Do not hard-code one formula for every model. Use the provider's current pricing and image-token documentation when you build a cost estimator.

Here is the practical shape to remember. The actual billed token count depends on the model, but larger images usually require more visual tokens until the provider resizes or caps them.

The difference between low and higher detail is large enough to design for. For "cat or dog?", low detail is enough. For "extract every line item and total from this receipt," use high detail, and consider cropping the receipt before sending it. Choose the detail level per task.

Cost Estimation Helper

Here is a utility function that estimates patch-based image tokens for GPT-5.5-style vision inputs before you send the image. Treat this as a planning estimate, not a billing source of truth.

Vision models are useful when the visual input changes the answer. The common production use cases fall into four categories.

1. Image Description and Captioning

You send an image, and the model describes it at the level of detail you request. This is useful for accessibility, catalog enrichment, moderation triage, and quick indexing of visual assets.

2. OCR and Text Extraction

Vision-language models can read text in context, which makes them useful for receipts, screenshots, forms, labels, and diagrams. They are often better than classic OCR when layout and semantics matter. They are not consistently better at raw character accuracy, especially for dense documents, tiny text, serial numbers, or regulated extraction workflows. For high-stakes OCR, compare against a dedicated OCR engine and reconcile disagreements.

3. Structured Data Extraction

Structured extraction turns vision models into application components. The goal is to return the fields your system needs, with uncertainty made explicit.

4. Diagram and UI Interpretation

Vision models are also useful for interpreting diagrams, flowcharts, wireframes, dashboards, and UI screenshots. They can explain what an error dialog says, identify visible layout problems, or turn a diagram into a rough text description. Treat the output as an interpretation.

Vision models have predictable failure modes. Handle them before exposing model output to users or downstream automation.

Counting

Ask a vision model to count overlapping objects and you may get a plausible but wrong number. Counting fails most often when objects overlap, are partially hidden, or look similar. If counts matter, use object detection, segmentation, or domain-specific measurement code.

Spatial Reasoning

Models have a rough sense of layout, not pixel-level geometry. "Is the red box left of the blue box?" is usually fine. "Which of these 10 marks is closest to the top-right corner?" is much less reliable. Crop regions, add coordinates, or use computer vision algorithms when precision matters.

Small Text

Even in high-detail mode, small text can be missed or misread. Fine print, low-contrast labels, angled photos, and compressed screenshots are common failure cases. Crop, deskew, enhance contrast, and enlarge the relevant region before sending it.

Hallucination

Hallucination is the failure mode to plan around most carefully, because the output can look correct. A model can produce text or objects that are not visible in the image. A blurry receipt can lead to a plausible but wrong total. A partially obscured serial number can be "completed" with invented digits. Prompt the model to mark uncertainty, but do not rely on prompting alone.

Math from Images

If an image contains a table of numbers, the model may extract the numbers correctly and then calculate incorrectly. Extract values into structured fields, validate them, and do arithmetic in application code.

Defensive Coding for Vision

Given these limitations, production systems should make uncertainty observable:

Confidence fields are not calibrated probabilities, but they are useful routing signals. Combine them with schema validation, consistency checks, OCR comparison, and human review queues for low-confidence cases.

The receipt processor below combines schema design, preprocessing, and validation into a complete pipeline. It takes a photo of a receipt, extracts structured data, handles poor image quality, and validates the results.

Step 1: Define the Schema

Two design decisions matter here. The model_validator cross-checks the total against the sum of line items, but it flags the inconsistency rather than rejecting the extraction. The model might have read the total correctly while missing or misreading a single line item, and rejecting the whole extraction would discard useful data. The image_quality_notes field lets the model communicate uncertainty, which the application can use to decide whether to route the receipt to a human review queue.

Step 2: Build the Extractor

Step 3: Add Preprocessing and Error Handling

Step 4: Run It

The pipeline handles the full lifecycle: input validation, image preprocessing to optimize token usage, structured extraction with schema validation, cross-field consistency checks, and graceful error reporting. It fails gracefully and communicates uncertainty back to the caller, which is what production systems need beyond the happy path.

References