Writing
Computer Use Is the Missing Layer Between Models and Software
Most integrations assume useful software exposes the right API. Much of real software never did.
Notes
Most AI integrations still assume that useful software exposes the right API.
The problem is that a large part of real software was never designed for models. Many desktop tools expose only part of their functionality through APIs. Older applications may be accessible only through the graphical interface people already use.
Anthropic's new Computer Use capability points at a different solution: let the model operate that interface directly.
I do not think this makes APIs obsolete. I think it adds the missing fallback layer between models and software that was built for humans.
APIs are clean because someone designed the contract
Tool use works well when software offers a narrow machine-readable contract.
A function such as export_asset(format, path, preset) is easier for a model to call than navigating an export dialogue. The parameters can be validated, permissions restricted, and errors returned as structured data.
That is why I would still prefer APIs for reliable production automation.
But software interfaces are much broader than their APIs. Creative applications are an obvious example. Blender has an extensive Python API, yet an artist's actual workflow still includes viewport state, temporary selections, modal tools, add-ons, file dialogues and visual judgments. Other production applications expose much less automation surface.
The GUI is the one interface nearly all of them share.
Anthropic's October 22 release gives Claude 3.5 Sonnet a computer-use tool that can inspect screenshots, move a cursor, click and type. Anthropic says the model was trained to translate a user instruction into sequences of actions against ordinary software rather than requiring developers to build a custom tool for every operation. That inversion is what interests me.
Instead of adapting every application to the model, the model begins adapting to existing applications.
A graphical interface can become an interoperability layer
A screen exposes enough state for a person to identify controls, understand the current step and choose an action without knowing the application's backend.
A vision-language model can attempt the same loop:
observe the screen → infer state → choose an action → execute it → observe the result
Anthropic describes Claude's implementation in almost literal terms. The model receives screenshots, estimates pixel coordinates for the target, and issues mouse or keyboard commands. It can then inspect the changed screen and continue.
It is less elegant than an API call, but far more general.
If the approach improves, an AI system does not need a dedicated integration for every application before it can attempt useful work. Any software with a visible interface becomes, at least in principle, an addressable environment.
That changes the economics of automation.
A computer-using model can cover gaps where integration code does not exist.
The interface is not the source of truth
There is a reason I call GUI control a fallback layer rather than the preferred one.
A screen is a lossy view of application state.
A button can move. A dialogue can cover another control. A notification can appear and disappear. Two visually similar controls can produce different side effects. The model may see only the current screenshot rather than the internal object graph, file state or transaction status behind it.
Anthropic is open about these limitations. Computer Use is in public beta, and the company describes it as slow and error-prone. Claude still struggles with basic interactions such as dragging, zooming and scrolling. Its view of the computer is closer to a sequence of screenshots than a continuous video stream, which means short-lived interface state can be missed.
The benchmark numbers show how early this is. On OSWorld, a benchmark containing 369 tasks across real desktop and web applications, Claude 3.5 Sonnet reaches 14.9% in Anthropic's screenshot-only setup and 22% when allowed more steps. The original OSWorld paper reports human performance above 72%.
That gap matters.
The interesting result is not that Claude can replace a person operating a desktop in October 2024. It cannot. The interesting result is that a general model can already perform some operations in previously unseen software using the same surface a person uses.
This changes how I think about creative software automation
Creative production is full of workflows that sit between structured automation and visual interaction.
In Blender, I can automate a large amount through Python. That is the right interface for deterministic tasks such as scene validation, batch export, object naming or render configuration.
But imagine an automation that has to cross Blender, a file browser, a web-based asset manager and another desktop application. Each system may expose a different API, authentication model or scripting environment. Some steps may expose none.
Computer use gives the model another route through the pipeline.
I would not let it blindly manipulate production files. I would combine interfaces according to their reliability. Use an API or script when one exists. Use structured project data for state. Use the graphical interface for the step that cannot be reached another way. Then validate the result before continuing.
The model becomes an orchestration layer that can move between machine interfaces and human interfaces depending on what the software exposes.
Reliability becomes a state problem
GUI agents make one issue from ordinary automation much harder: knowing whether an action actually succeeded.
A model can click Save. That does not mean the file was saved.
A confirmation dialogue, invalid path or misplaced click can leave the system in a different state than the model assumes.
For production systems, every meaningful action needs an observable completion condition.
If the model exports an asset, check whether the file exists and whether it matches expected properties. If it changes an application setting, inspect the resulting state rather than trusting the click. If an action is destructive, require confirmation or move it behind a more constrained tool.
The problem is no longer generating a plausible next action. It is maintaining verified state over a sequence of actions.
Security belongs in the same discussion. Anthropic points to prompt injection as a new risk because the model can read untrusted content from the screen and then take actions on the computer. A malicious page can become both data and instruction unless the system can separate the two reliably.
The more applications an agent can reach, the more carefully permissions need to be scoped.
My prediction: software becomes operable before it becomes integrated
I do not expect every software vendor to rebuild its product around AI agents.
Many applications will never expose every useful operation through clean machine-readable interfaces.
Computer use offers another path. Models may become capable of operating software before that software has any native AI integration.
If reliability improves, I expect agent architectures to use a hierarchy of interfaces. Direct functions and APIs will remain the first choice because they are fast, typed and testable. Command-line tools and scripting interfaces can handle broader workflows. GUI control can sit underneath them as the compatibility layer for everything else.
Using screenshots and mouse coordinates where a reliable API exists would be a downgrade. Their value is crossing the unautomated gap.
This matters in production environments, which accumulate old utilities, vendor tools and internal applications that may never receive a polished integration API.
A model that can understand those interfaces reduces the amount of custom glue needed before automation can begin.
We are not at the reliable version of that system yet. A 14.9% OSWorld score makes that clear.
But I think the architectural direction matters more than today's completion rate.
APIs let models use software that was prepared for them.
Computer use could let them work with the rest.
Sources
- Anthropic, Introducing computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku, 22 October 2024.
- Anthropic, Developing a computer use model, 22 October 2024.
- Xie et al., OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments, April 2024.
More
Other write-ups
Approval Is Not Publication
Thirty seven items in the queue, one approved, nothing published. The last arrow in the diagram is the only one that pays.
Neural Rendering Is Crossing From Reconstruction Into Synthesis
Reconstruction filled in what sparse sampling missed. DLSS 5 generates appearance the renderer never computed.
The Renderer Is Becoming a Training Data Engine
The renderer used to sit at the end of the pipeline. Physical AI gives the same scene a second job: teaching a model.
Local AI Is Becoming a Compute Fabric
Local AI meant one model on one machine. Routing inference across the devices already on a network changes the unit.
Agent Infrastructure Is Becoming a Product Category
Every team used to build the loop, the store, the sandbox. That layer is being sold rather than written.
Choosing a local model with a stopwatch, not a benchmark
Three models, five hard tasks, code actually executed. The official build was 2.4 times faster and more accurate than a community repack of the same model.
We measured real time lighting against baked light, and baked won
A 3D product simulation that had to look like an offline render. The real time version ran at 60 frames per second and looked like clay. Here is the measurement and the architecture that replaced it.
What actually broke in an agency run by agents
Three failures from a delivery stack that runs on AI. None of them were the model's fault, and all three reported success while producing nothing.
A Task Without A Check Command Is Not Automated
If a task has no command that can fail, the pipeline advances on the appearance of work.
A Gate The Model Writes Is A Gate The Model Loosens
Three quality gates returned green while the work behind them was wrong, each for a different reason.
The Scoring Model Was Wrong And It Put The Worst Lead First
A weighted sum let one axis substitute for the other, so a company with money and no problem ranked in the top twenty.
Building Software Got Easy. Getting Value Out Of It Did Not
Aristo took weeks to build. Everything after the build is still in progress, and that gap is the whole story.
Capability Is Becoming an Operational Risk Surface
Safety questions used to be about the text. Once a model can act, the capability itself becomes something to operate.
The Scene Graph Is Becoming an API for AI
A scene graph exists for artists and software. Agents are becoming another consumer, and they need structure rather than pixels.
Animation Is Moving From Clips to Motion Priors
Authored keyframes and blended clips are giving way to asking which constraints define acceptable motion.
Materials Are Becoming Learned Programs
A material is texture maps, parameters and shader code. It is starting to become a small learned program that answers a rendering question.
Procedural Systems Are Expanding Beyond Geometry
Geometry Nodes started with a narrow name. Blender 5.2 puts physics, sound and object data through the same graph.
The Unit of AI Work Is Becoming the Task, Not the Turn
Chat taught us to think one turn at a time. Long-running agents make the task the thing that is scheduled, resumed and reviewed.
Game Engines Are Becoming Operating Systems for Worlds
Engines have been judged on what they render and simulate. The Unreal 6 roadmap points at operating a world rather than drawing one.
The Model Is Becoming a Replaceable Backend
Choosing a provider used to mean choosing an architecture. A stable interface makes replacement possible and evaluation makes it safe.
The Harness Is Part of the Capability
The same model behaves differently depending on context policy, tool design and execution feedback. That surrounding software is not neutral.
Physics Engines Are Becoming Trainable Components
A simulator predicts what happens next. A differentiable one can answer which parameter should change to stop the failure.
The Agent Needs an Environment, Not Just Tools
A search function and a database query were enough for short loops. Longer work needs a place to stand.
Coding Agents Are Becoming General-Purpose Computer Workers
Repositories were a friendly environment: text in, terminal actions, checkable results. That was a starting point, not a boundary.
Open Standards Outlive Model Generations
A year after the MCP bet, the argument can be checked against what happened rather than what was hoped.
Colour Management Is a Pipeline Contract
Blender 5.0 reads as better display options. Giving a file an explicit working colour space is an architectural change.
The Best Model May Be a Router, Not a Model
GPT-5 moves model selection inside the system. The interesting unit stops being which model and becomes which compute policy.
World Models Are Not Game Engines Yet
Genie 3 generates a navigable 720p world at 24 fps. Production work needs state you can inspect when something goes wrong.
Memory Is Becoming a System Capability
A follow-up to the long-context argument. Storing, selecting and expiring facts is turning into a named part of the product.
Coding Agents Change the Unit of Software Work
AI coding tools have been judged where code appears on screen. The boundary moves when the agent owns a task instead of a snippet.
Agents Need Protocols Between Each Other, Not Just Tools
Tool calling solves the inside of the loop. It says nothing about one agent reaching another built by a different team on another platform.
Agents Need a Runtime, Not a Prompt Loop
An agent is no longer well described as a prompt inside a while loop. The useful abstraction owns execution around the model.
Reasoning Is Not the Only Path to Better Models
Longer thinking improves maths and code. A model that solves a logic puzzle and misreads ordinary intent is not the better production model.
Rendering Is Becoming a Reconstruction Stack
Sparse samples, motion data and lower-resolution frames become a larger final result. Debugging becomes layered when reconstruction sits in the middle.
Agent Reliability Is an Evaluation Problem, Not a Prompting Problem
When an agent misses a step, the usual fix is a stricter prompt. The failure is more often in how completion is detected.
MCP Might Matter More Than Another Model Release
A model can reason well and still be useless inside a company if it cannot reach the files, repositories and tools where work lives.
Inference-Time Compute Is a New Scaling Axis
o1 improves when it is allowed to spend longer on a problem. A benchmark score without a compute budget is an incomplete number.
The Final Pixel Won't Come From the Renderer
Geometry, camera and scene structure stay reliable ground truth. More of final appearance is moving into learned systems.
Open Models Are Becoming Research Infrastructure
Llama 3.1 gets discussed as a benchmark result. The licence terms change which experiments are possible at all.
The Model Is Becoming a Runtime
Function calling, code execution and structured output turn inference into a loop. The model stops being a text generator and starts being a control layer.
Multimodality Changes the Architecture, Not Just the Interface
GPT-4o is easy to read as a faster interface. Training one model end to end across text, vision and audio is an architectural change.
Benchmark Scores Are Not Model Capability
Claude 3 posts 86.8% on MMLU and 50.4% on GPQA Diamond. The chart is useful and it is not the same thing as capability.
Long Context Is Not Memory
Gemini 1.5 makes a million tokens usable. A larger working set is not a system that decides what should survive the session.