Ask a chatbot to file an expense report and it will cheerfully list the steps: log in, click New Expense, upload the receipt, hit Submit. Ask a computer use agent and it does the clicking. That gap between describing an interface and operating one is where a lot of the practical work in AI has moved over the past eighteen months.
Anthropic shipped a beta of computer use in Claude in October 2024. OpenAI followed with Operator in January 2025, running on a model it calls CUA. Google built the same capability into Gemini as Project Mariner, developed inside the same research lab that produced AlphaFold. All three do the same basic trick: look at a screenshot, decide on a click or a keystroke, take it, then look again.
That loop sounds trivial. In practice it is the difference between software that needs an API and software that just needs a login.
What separates a computer use agent from ordinary automation
Robot process automation has existed for years. Tools like UiPath drive browsers by latching onto element IDs, CSS selectors and fixed coordinates. They are fast, cheap and utterly brittle. Move a button 40 pixels and the script dies. Put a modal in the way and it dies. Show it a desktop application wrapped in a Citrix window and it has nothing to grip.
Computer use agents work from pixels instead. They see what a human sees, which means they generalise to any interface the model can read: a supplier portal from 2011, a legacy ERP system, a PDF viewer, a virtual desktop. No integration project, no vendor API request, no six-week procurement cycle.
Two things had to improve before this became viable. Vision models got good enough to parse a messy, badly labelled interface. Reasoning models got good enough to recover when a plan goes sideways. If the underlying mechanics of those models are still fuzzy, our plain-English guide to how AI actually works covers the model side; the agent side is mostly plumbing wrapped around it.
Inside the loop
Perception
The agent gets a screenshot, usually scaled to something like 1280 by 800 pixels, and predicts the coordinates of the next click. Some implementations also feed in the page’s accessibility tree or DOM, which improves accuracy on standard websites. Pure vision generalises further, because it does not care whether the thing on screen is a web page, a spreadsheet or a Java applet.
Action
The available moves are deliberately boring: click at a point, type text, press a key combination, scroll. That narrow set turns out to be enough for almost everything a knowledge worker does on a screen. Agents run inside a containerised browser or a throwaway virtual machine, so a stray click lands on a disposable desktop rather than your finance system.
The step budget
Every action requires a fresh screenshot and a fresh model call. At two to ten seconds per step, a forty-step task takes several minutes and costs real money. The economics matter. OpenAI’s 2025 lineup prices a long agentic run very differently from a single chat reply, and the same arithmetic applies to every vendor in this space. Agents that wander in circles are expensive agents.
Where they earn their keep right now
The strongest deployments share a profile: high volume, low glamour, a clear definition of done.
- Legacy portals with no API. Insurance claims, customs filings, government registries, freight bookings. The interface is ugly and slow, and nobody is going to build an integration for it.
- Invoice and order reconciliation. Pull a PDF from an inbox, read three fields, key them into an accounting system, flag the mismatch.
- Software testing. An agent that clicks through a checkout flow like a bored customer finds broken buttons that unit tests never touch.
- Vendor onboarding. Filling the same twelve-field registration form across forty supplier sites, each with its own quirks.
- Logged-in research. Pulling pricing or stock levels from competitor sites that hide everything behind an account.
The benchmark reality check
OSWorld, the standard academic test for computer use, gives agents a real desktop and asks them to complete multi-step tasks in real applications. Humans score around 72 percent. The best agents sat in the 40s for much of 2025, with the strongest systems edging higher as new models landed. WebArena tells a similar story on web-only tasks.
Read those numbers plainly: the best agent in the world fails a substantial share of tasks a competent temp would sail through. Our roundup of the best AI agents judged by what they actually finish rather than what they demo is a useful antidote to launch-day enthusiasm.
How they fail, specifically
Failure modes in this space are consistent enough to design around.
Drift on long tasks. A ten-step task works. A sixty-step task accumulates small errors until the agent is confidently editing the wrong record.
Visual ambiguity. Two identical ‘Continue’ buttons, a date field that wants DD/MM and a model that assumes MM/DD, a dropdown that needs a scroll to reveal the right option.
Authentication walls. Two-factor codes, captchas, session timeouts. Most production setups hand these back to a human rather than pretending to solve them.
Silent success. The worst outcome is not a crash. It is an agent that reports the invoice was approved when it actually saved a draft. Verification steps matter more than the agent itself, which is the core argument in how to build AI agents that actually finish the job.
Prompt injection. Any page the agent reads can contain text aimed at the model rather than the human, such as a hidden instruction to forward data somewhere. Once an agent has your session and your inbox, a poisoned web page is a live attack surface, not a theoretical one. Anthropic flagged this class of risk the day it launched.
Permissions are the real design problem
Most teams spend their first month tuning prompts and their second month realising the hard part is authority. An agent with a browser and a login is, functionally, an employee with your credentials.
A few rules that hold up in practice. Give the agent a dedicated account with the minimum permissions the task needs, never your own. Run it in an isolated browser profile with no personal cookies, password manager or single sign-on session. Put a human approval gate in front of anything that spends money, deletes a record or sends a message to an outside party. Keep a full screenshot trace of every run, because when something goes wrong you will need to watch it happen.
Pick a boring first task and measure it properly
The teams getting value from AI computer use agents are not the ones with the most ambitious roadmaps. They are the ones who picked one dull, repetitive workflow, ran it fifty times, and counted.
Choose a task with a machine-checkable finish line. ‘Create the purchase order’ is checkable: either the record exists with the right values or it does not. ‘Research this market’ is not. Run fifty real instances, log the straight-through completion rate, and count how many minutes a human spends cleaning up the failures. If the answer is 80 percent completion plus two minutes of supervision on the rest, you have a workflow worth scaling. If it is 45 percent plus twenty minutes of repair, you have a demo.
Then expand sideways, one adjacent task at a time, reusing the same permissions, the same sandbox and the same verification checks. The compounding happens in the operational scaffolding, not the model version. And keep the human in the loop for anything that leaves a trace outside your own systems, because the agents are genuinely good now at the clicking, and still uneven at knowing when they should stop.