What happens when the system you need to interact with does not have an API?

Plenty of business-critical software lives behind a GUI and nothing else. Think legacy ERP systems, internal admin dashboards, government portals with no public endpoints, or desktop applications that were never designed for automation. Your agent can reason, plan, and call tools all day, but if the only way to submit a form is to click a button on a web page, it is stuck.

This is where computer use agents come in. Instead of calling structured APIs, these agents interact with software the same way a human does: they look at the screen, decide where to click or type, execute that action, and then observe what changed. It sounds simple, but it opens up an entirely different class of automation.

Suddenly your agent can fill out forms, navigate multi-step workflows, extract data from dashboards, and test user interfaces, all without a single API integration.

The trade-off is that screen-based interaction is slower, more fragile, and harder to get right than API calls. Pixels shift, layouts change, and a button that was at coordinates (450, 320) yesterday might be at (460, 335) today. Building reliable computer use agents requires understanding the perception-action loop, the tools available, and when this approach actually makes sense versus just building a proper integration.

At the core of every computer use agent is a perception-action loop. It works like this: the agent captures what is currently on the screen, sends that image (or a structured representation of it) to a model, the model decides what action to take, the agent executes that action on the computer, and then the cycle repeats. The agent keeps looping until it determines the task is complete.

Compare this to the agent loop. The structure is the same, observe-think-act, but the observation is visual rather than textual, and the actions are physical UI interactions rather than function calls. This makes the loop inherently noisier. An API returns clean JSON. A screenshot returns millions of pixels that the model has to interpret.

Each iteration is also more expensive. The model needs to process an image on every cycle, which uses significantly more tokens than processing a text-based tool result. A typical screenshot at 1280x800 resolution might cost 1,000-2,000 tokens depending on the model. If your loop runs 15 iterations, that is 15 screenshots plus the growing conversation history. Keep this cost in mind when designing computer use workflows.

The reliability challenge is real too. The model might misidentify a button, click the wrong element, or misread text in the screenshot. Each mistake compounds because the agent is now in a different state than it expected.

Good computer use agents need robust error recovery: the ability to recognize when something went wrong and course-correct, rather than blindly continuing down a wrong path.

There are two fundamentally different ways for a computer use agent to "see" what is on screen. Each comes with distinct trade-offs.

Screenshot-based (pixel) approaches send a raw image of the screen to the model. The model interprets the visual layout, reads text via its vision capabilities, identifies interactive elements, and decides where to click. This is the most general approach because it works on any application, web, desktop, or even terminal UIs. If a human can see it, the model can (in theory) see it too.

DOM-based approaches work specifically with web browsers. Instead of (or in addition to) taking a screenshot, the agent extracts the page's DOM (Document Object Model), the underlying HTML tree structure. This gives the agent structured, machine-readable information: element types, labels, attributes, text content, and whether elements are visible or interactive. The agent can reference elements by their CSS selector or accessibility label rather than by pixel coordinates.

In practice, the best agents combine both. They use DOM parsing for structured information (what elements exist, what their labels are, which ones are clickable) and screenshots for visual context (what the page actually looks like, spatial layout, confirmation that the right page loaded). This hybrid approach gives the model the best of both worlds: structured data for precise element targeting and visual data for situational awareness.

One important nuance: DOM parsing does not work for desktop applications. If you need to automate a native Windows app, a macOS dialog, or anything that is not a web browser, you are limited to screenshot-based approaches (or OS-level accessibility APIs, which we will cover later). This is one reason screenshot-based models like Claude Computer Use are significant. They can handle any application, not just web pages.

Anthropic's Claude Computer Use is one of the first production-ready implementations of a computer use agent. It gives Claude the ability to see a screen and control a computer through a defined set of tools. Understanding how it works gives you a practical model for building your own computer use systems.

The API works by defining three tool types that Claude can invoke during a conversation:

When you send a message to Claude with these tools enabled, Claude can respond with tool use requests just like regular function calling. The difference is that the tools map to physical computer actions rather than API calls.

Here is what a basic setup looks like:

The key architectural idea here is that Claude treats computer interaction as just another set of tools. It does not need special "computer vision mode." It uses the same function calling loop you already know from Module 6, but the functions happen to control a mouse and keyboard instead of querying a database.

After every action, the agent takes a new screenshot and sends it back to Claude. This is the "observe" step. Claude looks at the updated screen, reasons about whether the action succeeded, and decides what to do next. If a click landed in the wrong place or a page did not load as expected, Claude can see that and adjust.

A few practical things to know about Claude Computer Use. The display resolution you specify in the tool definition matters. Claude uses it to map pixel coordinates. If you say the display is 1280x800 but your actual screenshots are 2560x1600 (Retina), the coordinates will be off by a factor of 2. Always match the tool configuration to your actual screenshot resolution. Also, running this in a sandboxed virtual machine or container is strongly recommended. You are giving an AI model the ability to click and type on a computer. A sandboxed environment limits the blast radius if something goes wrong.

For web-specific automation, you often do not need full computer use. Browser automation libraries like Playwright give you programmatic control over a browser with a much more reliable interface than pixel-based clicking. When your target is a web application, Playwright-based agents are usually the better choice.

Playwright lets you launch a browser, navigate to URLs, interact with page elements using CSS selectors, extract content, and take screenshots. Combined with an LLM for decision-making, you get an agent that can navigate complex web workflows without needing to interpret raw pixels.

This approach is DOM-based. Instead of sending a screenshot, we extract the page's text and interactive elements as structured data. The LLM sees a list of clickable buttons and links with their labels, which is far easier to reason about than a raw image. The token cost per iteration is also much lower.

Notice how the agent uses Playwright's query_selector_all to find elements and fill or click to interact with them. These are reliable, deterministic operations. When the LLM says "click element 5," you know exactly which element that is. Compare this to pixel-based clicking where the model says "click at (450, 320)" and you hope the button is actually there.

The trade-off is that this only works for web pages. And even for web pages, some sites use heavy JavaScript rendering, iframes, or shadow DOM that make extraction tricky. Frameworks like Browser Use (built on Playwright) handle many of these edge cases and add features like automatic element annotation, screenshot fallbacks, and multi-tab management.

Web browsers are not the only thing you might need to automate. Desktop applications, system dialogs, file managers, and native apps all require a different approach. Desktop automation operates at the OS level, interacting with windows, menus, and UI elements through accessibility APIs or direct input simulation.

On macOS, the primary tool is the accessibility framework (via pyobjc or AppleScript). On Windows, you have UI Automation through libraries like pywinauto. On Linux, xdotool and AT-SPI handle the job. For cross-platform work, pyautogui provides a simpler but less reliable option that works by simulating mouse and keyboard input at the OS level.

This code is fragile. It uses fixed delays (time.sleep), types at specific intervals, and assumes the UI responds exactly as expected. In real desktop automation, you need to combine this kind of input simulation with screen observation to verify each step succeeded.

The more robust pattern looks like this:

The key difference from web automation is that desktop automation lacks the equivalent of a DOM. There is no clean tree structure to query. You are working with accessibility trees (which vary wildly in quality between applications), pixel matching, or OCR. This is exactly where screenshot-based models like Claude Computer Use shine, because they can handle any visual interface without needing structured element data.

For production desktop automation, consider running agents inside virtual machines or containers with virtual displays. Tools like Xvfb on Linux give you a headless display that the agent can interact with without needing a physical monitor. This makes it possible to run multiple agents in parallel and isolates them from your real desktop environment.

Not every problem needs a computer use agent. In fact, most do not. The decision of whether to use a traditional API integration or a computer use agent is one of the most important architectural choices in this space.

Here is a decision framework:

The general rule: use an API when one exists and meets your needs. Fall back to computer use when there is no API, the API does not cover your use case, or the cost of building an API integration exceeds the cost of a less reliable screen-based approach.

A common real-world pattern is using computer use agents for legacy system integration. Many enterprises run critical processes on software built in the 1990s or 2000s that has no API and never will. Replacing the software is a multi-year project. A computer use agent that fills forms and clicks through workflows in the legacy system can bridge the gap while a proper migration happens. It is not pretty, but it works.

Other strong use cases include:

The weakness of computer use agents is reliability at scale. A human can tolerate a page that loads slightly differently today. An agent that expects a specific button at a specific location might fail. Building robust agents means adding retry logic, visual verification, and fallback strategies for when the UI does not match expectations.

References