Some systems still need to be automated through the screen.

Many business workflows still live in GUIs: legacy ERP screens, internal admin dashboards, government portals, bank websites, virtual desktops, and desktop applications that were never designed for automation. A model can plan and call APIs, but if the only way to finish the work is to click a button in a browser or native app, an API-only agent has no useful path forward.

Computer use agents operate through the user interface. They inspect the current screen, choose a mouse or keyboard action, execute it, and observe the result. The interface is no longer a JSON schema; it is the same visual environment a person would use.

That makes them useful for form filling, GUI regression testing, legacy system integration, dashboard extraction, and workflows that cross several applications without a shared backend.

The trade-off is real. Screen automation is slower, more expensive, and more fragile than API integration. Layouts shift, modals appear, focus moves, network delays change timing, and a coordinate that worked yesterday may click the wrong element today. A reliable system wraps the model in verification, recovery, permissions, logging, and hard limits.

Every computer use agent runs an observe-and-act loop. It captures the current UI state, gives the model either an image or a structured representation of that state, receives the next action, executes it, and observes again. The loop stops when the task is complete, a guardrail blocks the action, or the run hits a budget or step limit.

The structure is the same observe-think-act loop used by other agents, but the observation is much noisier. An API call returns typed fields. A screenshot returns pixels. The model has to infer text, layout, clickable controls, disabled states, and current focus from what is visible.

Each iteration is also more expensive. Image inputs usually cost more than short text tool results, and UI tasks often need many observations because every click, page load, and error state must be checked. Cap the step count, cap the spend, and use lower-resolution or cropped observations when the task allows.

Failures compound. The model may click the wrong control, type into the wrong field, miss a toast notification, or misread a disabled button as active. After one bad action, the agent may no longer be in the state it expected, so later actions can affect the wrong record, page, or tab.

A reliable agent verifies each important action and has a clear recovery path. It checks whether the expected page appeared, whether the value was entered, whether the submit button became enabled, and whether the final confirmation matches the task. Continuing after a failed UI action can corrupt data in the underlying system.

There are two common ways for a computer use agent to understand what is on screen. Each has different strengths.

Screenshot-based approaches send an image of the screen to the model. The model reads visible text, interprets layout, identifies controls, and returns an action such as click, drag, type, scroll, or press a key. This is the most general approach because it can work across web pages, native applications, remote desktops, and terminal UIs. Generality has limits: hidden state, tiny controls, low contrast, virtualized lists, and stale screenshots can still confuse the model.

DOM-based approaches work in browsers. Instead of relying only on pixels, the agent extracts page structure: element types, accessible names, roles, labels, attributes, text content, and visibility. The agent can target an element by selector, role, or accessibility label instead of raw coordinates.

Production browser agents usually combine both. They use the DOM or accessibility tree for precise targeting and screenshots for layout, visual confirmation, and cases where the DOM does not explain what the user can actually see. The model gets enough context to choose a sensible action, and the runtime uses deterministic selectors whenever it can.

DOM parsing does not help with native desktop applications, remote desktops, or virtualized enterprise apps. In those environments you rely on screenshots, OCR, and OS accessibility APIs. Accessibility trees are valuable when available, but their quality varies by application. Some enterprise apps expose rich semantics; others expose little more than a bitmap.

Modern model providers expose computer use through tool interfaces. The details vary by provider. OpenAI's current API includes a computer tool for computer-use workflows. Anthropic's computer use tool is still documented as a beta feature with versioned tool identifiers and beta headers. Those names change over time, so treat exact model names, tool types, and opt-in headers as deployment details to check against the provider docs.

The architecture is more stable than the API names: the model asks for a UI action, your application decides whether that action is allowed, executes it inside a controlled environment, and returns the next observation.

Common capabilities look like this:

Treat these as separate privilege levels. A browser-only workflow may need a computer or browser tool. A coding workflow may need file editing and shell execution. Many business workflows should not receive shell access at all.

Here is the shape of a minimal Anthropic-style loop. This example is intentionally small and macOS-specific: it uses screencapture for screenshots and cliclick for mouse and keyboard actions. In production you would usually run the same idea inside a Linux VM or container with a virtual display, use the provider's current tool version, and add stricter action validation.

Computer interaction is still tool use. The model requests an action, the runtime validates it, performs it in a controlled environment, and returns an observation. The same engineering disciplines apply as with any other tool: schema design, permission checks, logging, retries, timeouts, and clear stopping conditions.

After every action, the agent takes a new screenshot and sends it back to the model. This is the observe step. The model looks at the updated screen, checks whether the action succeeded, and decides what to do next. If a click landed in the wrong place or a page did not load as expected, the next observation gives the agent a chance to recover.

Several practical details matter. The display resolution in the tool definition must match the screenshot coordinate space. Retina scaling, browser zoom, remote desktop scaling, and CSS zoom can all produce off-by-two or off-by-fraction errors. Keep the environment deterministic: fixed viewport, fixed zoom, predictable fonts, stable locale, and no personal accounts or secrets available to the agent. Run computer-use agents in a sandboxed VM, a container with a virtual display, or a disposable browser profile. The model is controlling a real machine, so the environment must be isolated enough that a wrong click cannot reach data outside the sandbox.

For web automation, full computer use is often the wrong default. Browser libraries such as Playwright provide deterministic control over navigation, selectors, form filling, screenshots, tracing, downloads, and network events. When the target is a web application, a Playwright-based agent is usually cheaper and more reliable than pixel-level clicking.

Playwright lets you launch a browser, navigate to URLs, interact with page elements using CSS selectors, extract content, and take screenshots. Combined with an LLM for decision-making, you get an agent that can navigate web workflows without asking the model to reason entirely from pixels.

This approach is DOM-based. It still runs the same observe-think-act loop as the screenshot agent earlier, but the observation is a list of structured elements rather than a base64 image. The model sees controls with labels and metadata, which is easier to reason about than raw pixels and cheaper to send through the context window.

The model chooses a semantic action, but Playwright performs the interaction. When the model says "click element 5," the runtime maps that index to a concrete visible DOM element and calls click(). This is safer than asking the model to click coordinate (450, 320), because the element can still be found even if the page shifts by a few pixels.

The trade-off is coverage. DOM extraction only works in browsers, and even then you must handle iframes, shadow DOM, canvas-heavy apps, virtualized lists, popups, downloads, authentication flows, and bot defenses. Frameworks such as Browser Use build on Playwright and add element annotation, screenshot fallbacks, multi-tab handling, and higher-level browser state management.

Web browsers are not the only thing you might need to automate. Desktop applications, system dialogs, file managers, and native apps require a different approach. Desktop automation operates at the OS level, interacting with windows, menus, and UI elements through accessibility APIs or direct input simulation.

On macOS, the main option is the accessibility framework through tools such as pyobjc or AppleScript. On Windows, UI Automation is available through libraries such as pywinauto. On Linux, common options include xdotool and AT-SPI. For cross-platform work, pyautogui is a simple starting point because it simulates mouse and keyboard input at the OS level. It is also less reliable because it has little understanding of which window is focused or what a control means.

This code is fragile. It uses fixed delays (time.sleep), types at specific intervals, and assumes the UI responds exactly as expected. Production desktop automation combines input simulation with screen observation or accessibility checks so it can verify each step before moving on.

A more reliable pattern looks like this:

The difference from web automation is the quality of structure. Desktop automation may have an accessibility tree, but it is not as consistent as a browser DOM. Some applications expose names, roles, values, and actions. Others expose little useful metadata. Screenshot-based models are useful here because they can reason over the visible interface when structured UI data is incomplete.

For production desktop automation, run agents inside virtual machines or containers with virtual displays when possible. Tools such as Xvfb on Linux give an agent a display it can interact with without using a physical monitor. This makes it easier to run multiple agents in parallel and keeps them away from your real desktop environment.

Not every problem needs a computer use agent. Most do not. The decision between an API integration, browser automation, and full computer use is one of the most important architectural choices in this space.

Here is a decision framework:

In practice: use an API when it exists and covers the workflow. Use browser automation when the target is web-based and selectors are stable enough. Use full computer use when the workflow only exists through a GUI, spans applications, or requires visual interaction that cannot be represented cleanly as a DOM operation.

A common enterprise pattern is legacy system integration. Many organizations still run critical processes on software that has no maintained API and will not be replaced soon. A computer use agent can bridge that gap by filling forms and extracting confirmations while the organization works toward a proper integration or migration. The healthiest use is short-term coverage during a migration window, not a permanent layer the rest of the system depends on.

Other strong use cases include:

The weakness is reliability at scale. A person can notice when a page renders differently and pause. A model-driven agent may misread the new state and continue with a stale plan. Systems that hold up in production add explicit verification, retries with limits, fallbacks to human review, idempotency checks, and audit logs for every action.

References