From Chatbot to Coworker: Teaching a Model to Use a Computer
For two years, the standard way to give an AI model hands was tool calling: you wrote a function, exposed it as a tool, and the model decided when to call it. It works beautifully when every tool is an API. It collapses the moment you need to work with software that has no API at all. The spreadsheet your finance team uses, the legacy CRM, the design tool, the internal portal — most of the software that actually runs a business was never built to be called by a machine.
Computer-use agents attack that problem from a completely different angle. Instead of teaching the model about your APIs, you give it a screen, a mouse, and a keyboard. The model looks at a screenshot, decides what to click, where to type, and what to press, then observes the result and repeats. It is the same loop a human follows when using a new application, and it is why 2026 has become the year of the computer-use agent: from OpenAI's GPT-6 Astra driving Blender to the always-on teammates from xAI and Anthropic, the hottest capability in AI is simply the ability to operate existing software like a person would.
The best way to understand this wave is not to watch demos but to build one. In this tutorial, you will create a minimal computer-use agent in Python that can look at your desktop, perform actions, and complete a small real task — with full visibility into every step. You will need about an hour and a computer running Windows, macOS, or Linux.
How the Loop Works
Every computer-use agent, from the cheapest open-source script to the most expensive flagship product, runs the same four-step loop:
- Observe: capture a screenshot of the screen (or a window) and send it to the model.
- Decide: the model responds with a tool-use request naming an action, such as click, type, scroll, or press a key.
- Act: your code executes that action on the real screen with a mouse-and-keyboard library.
- Repeat: you send back a fresh screenshot so the model can see the result of its action and plan the next one.
The model never sees your screen directly. It sees a compressed image, chooses from a small vocabulary of actions, and relies on you to execute them and report back. That separation is what makes the system safe to build: you control exactly which actions are permitted, you can watch every step, and you can stop the loop at any moment.
Step 1: Set Up Your Environment
Create a fresh virtual environment and install three packages: the Anthropic SDK for model access, PyAutoGUI for mouse and keyboard control, and Pillow for image handling. If you prefer OpenAI or another provider, the structure is identical — swap the client and the tool schema, and the loop below still applies.
mkdir computer-agent cd computer-agent python3 -m venv venv source venv/bin/activate pip install anthropic pyautogui pillow export ANTHROPIC_API_KEY=your-key-here
On macOS you will also need to grant your terminal Accessibility permission so PyAutoGUI can control the mouse and keyboard; on Windows and Linux, no extra permission is required for a local session. Run every experiment in a window you do not mind being clicked around in — you are about to give a model control of your cursor.
Step 2: Write the Screen and Action Helpers
First, a helper that captures the screen and returns base64-encoded PNG data, and an executor that maps the model's requested actions onto PyAutoGUI calls. Keep the display size fixed and match it in the tool definition so the model's coordinates line up with your pixels.
import base64, io, time
import pyautogui
from PIL import Image
W, H = 1920, 1080 # match your display size
def grab_screen():
img = pyautogui.screenshot()
img = img.resize((W, H))
buf = io.BytesIO()
img.save(buf, format="PNG")
return base64.b64encode(buf.getvalue()).decode()
def act(name, args):
if name == "screenshot":
return grab_screen()
if name == "click":
x, y = args["coordinate"]
pyautogui.click(x, y)
elif name == "type":
pyautogui.write(args["text"])
elif name == "key":
pyautogui.press(args["key"])
elif name == "scroll":
pyautogui.scroll(args.get("delta", -200))
time.sleep(0.8) # let the UI settle
return grab_screen()
The helper returns a fresh screenshot after every action, which becomes the model's next observation. That single design choice — always feeding back the real, post-action screen — is what prevents the agent from drifting into a fantasy of what the screen should look like.
Step 3: Build the Agent Loop
Now the core loop. We declare a computer tool with the display dimensions, send the first screenshot along with the task, and then keep executing tool calls until the model produces a plain text reply instead of another action.
from anthropic import Anthropic
client = Anthropic()
TOOLS = [{
"type": "computer_use",
"name": "computer",
"display_width_px": W,
"display_height_px": H,
}]
def run(task, max_steps=10):
messages = [{"role": "user", "content": [
{"type": "image", "source": {"type": "base64",
"media_type": "image/png", "data": grab_screen()}},
{"type": "text", "text": task},
]}]
for step in range(max_steps):
response = client.messages.create(
model="your-computer-use-model-id",
max_tokens=1024,
tools=TOOLS,
messages=messages,
)
acted = False
for block in response.content:
if block.type == "tool_use":
acted = True
image = act(block.name, block.input)
messages.append({"role": "assistant",
"content": [block]})
messages.append({"role": "user", "content": [
{"type": "tool_result",
"tool_use_id": block.id,
"content": [{"type": "image", "source": {
"type": "base64", "media_type": "image/png",
"data": image}}]}]})
elif block.type == "text":
print(block.text)
if not acted:
print("finished in", step + 1, "steps")
break
run("Open a text editor, type 'hello from my first agent', "
"and save the file to the desktop.")
Replace your-computer-use-model-id with a computer-use-capable model from your provider's current documentation. Everything else in the loop is provider-agnostic: the pattern of screenshot, decide, act, observe is the same whether the underlying model is Claude, GPT, or Gemini.
The model proposes; your code disposes. Because every action executes through a small whitelist of PyAutoGUI calls, you keep full veto power over what the agent is allowed to do — which is exactly how you should run it.
Step 4: Run, Watch, and Debug
Launch the script and watch closely. The agent will move your cursor, take over the keyboard, and narrate its reasoning as text. The most common failure modes are easy to diagnose:
- Wrong coordinates: if clicks land in the wrong place, the display size in the tool definition does not match the resize in grab_screen. Keep them identical.
- Stuck loops: if the agent repeats the same action, the screen probably did not change. Increase the settle delay, or add a guard that stops after N identical actions.
- Dialogs and popups: models often miss modal windows. Ask it to screenshot again after every action — the fresh observation usually fixes the confusion.
- Permission errors: on macOS, re-check Accessibility permissions for your terminal app.
A good first test is a completely offline task, like opening your system calculator and typing a calculation, so a network hiccup cannot interfere with your debugging.
Step 5: Harden It Like a Production System
Once the basic loop works, resist the urge to unleash it on your real inbox. Add the production guardrails that every serious computer-use deployment needs:
- Sandbox first: run experiments in a virtual machine or a dedicated user account before touching anything important.
- Approval gates: pause before destructive actions — sends, deletes, purchases — and require a human to confirm.
- Step budgets and timeouts: cap every run at a maximum number of actions and a wall-clock limit.
- Full logging: record every screenshot and action to disk so you can replay what the agent did and why.
- Least-privilege sessions: log the agent into a restricted account with access only to the applications the task requires.
Where to Go From Here
Your first agent is a skeleton, but it is the same skeleton that powers the billion-dollar products of 2026. From here you can extend it in four directions: add a browser-only environment (the safer cousin of desktop control, supported natively by several providers), give the agent persistent memory so it remembers how you like tasks done, swap the greedy loop for a planner that decomposes large jobs into subtasks, or plug in structured outputs so the agent reports its results as JSON your other systems can consume.
The deeper lesson is the one that explains the whole computer-use wave: the bottleneck in automation was never the model — it was access to the software. Give a capable model a screen and a whitelist of actions, and the long tail of un-API-able legacy tools finally becomes automatable. You just built the smallest possible version of that future. Now go point it at something tedious and watch it work.
