Computer use

Computer use gives an agent or a script a machine’s desktop the way a person at the screen has it: take a screenshot, decide what to do, then click, drag, scroll and type. It is the same desktop the Desktop view shows, so you can watch every step and step in. Use it for GUI apps that aren’t a web page, and for pages a browser tool can’t work with. For ordinary web pages, agent-browser is the better tool, because it reads and acts on the page itself rather than on pixels.

Try it

boxd machine new demo
boxd machine exec demo -- agent-browser open https://example.com   # something to look at
boxd machine computer demo screenshot -o screen.png
boxd machine computer demo click 640 400 --screenshot after.png

Open boxd machine desktop demo to watch it happen.

How it works

One action per call. Every call answers with the screen’s size and where the pointer is. Any action can also take a screenshot after itself in the same call, so a look-act-look loop costs one call per step.
Coordinates are screen pixels from the top left. A screenshot scaled down with --max-width or --max-height is smaller than the screen, so scale coordinates you read off it back up before clicking. The agent adapters do this for you.
The desktop starts on demand. The first action starts it if it isn’t running, which can take about 10 seconds on a machine that has just started. Later actions don’t wait for it.
Sleeping machines wake. A machine on standby or hibernated is woken by an action. A stopped machine isn’t started for you, so the call fails until you start it.
Who can use it. Anyone who can run commands on a machine: its owner, any member of the organization when the machine is shared, and API keys with access to it. Actions run as the machine’s user.

Computer use needs the desktop, so the note about machine images on the Desktop page applies here too.

From the CLI

The laptop CLI and the CLI inside a machine have the same commands. Inside a machine, leave out the name to act on that machine, or name another machine you can reach.

boxd machine computer myapp screenshot -o screen.png     # --format jpeg|webp, --quality, --max-width, --max-height, --crop X,Y,W,H, --delay
boxd machine computer myapp click 640 400                # --button right, --double, --triple, --count N, --modifiers ctrl+shift
boxd machine computer myapp move 100 100
boxd machine computer myapp mouse-down 100 100           # a drag step by step; also mouse-up
boxd machine computer myapp drag 100 100 400 300         # through more points: X Y X Y ...
boxd machine computer myapp scroll 640 400 --down 5      # wheel clicks; --up, --left, --right
boxd machine computer myapp type "hello world"           # `-` reads the text from stdin
boxd machine computer myapp key ctrl+l                   # Return, alt+Tab, ctrl+shift+t; --repeat N, --hold-ms MS
boxd machine computer myapp wait 1000                    # milliseconds
boxd machine computer myapp cursor --json                # {"screen": {...}, "cursor": {...}}

Every action takes --screenshot FILE and the same image options as screenshot:

boxd machine computer myapp click 640 400 --screenshot after.png --delay 500
boxd machine computer myapp key Return --screenshot - --format jpeg > after.jpg

The format defaults to the file’s extension, then PNG, and - is stdout. Key names are X key names, as xdotool takes them, plus common aliases such as enter, esc, cmd and pageup.

--crop picks the part of the screen. --region is the machine’s region, as on every machine command.

From the SDKs

machines.computer has the same actions. See the Python and TypeScript references for all of them.

c = boxd.machines.computer

r = c.screenshot("myapp", max_width=1280)
r.screenshot.data                                  # PNG bytes
c.click("myapp", 640, 400)
c.type("myapp", "hello")
r = c.key("myapp", "Return", screenshot=True)      # and a screenshot after it, same call
r.screen_width, r.screen_height, r.cursor

const computer = boxd.machines.computer;

const { screenshot } = await computer.screenshot("myapp", { maxWidth: 1280 });
screenshot.data;                                   // PNG bytes
await computer.click("myapp", { x: 640, y: 400 });
await computer.type("myapp", "hello");
const r = await computer.key("myapp", "Return", { screenshot: true });  // and a screenshot after it, same call
r.screenWidth, r.screenHeight, r.cursor;

From any other language, call ComputerAction on the gRPC API.

Hold the screen

By default the screen follows whoever is watching it: a desktop nobody has opened is 1280x800, and while someone has the Desktop view open, the screen is the size of their window. That can change the screen’s size in the middle of an agent’s work, and coordinates it worked out a moment ago then point at the wrong place. An agent should hold the screen while it works:

boxd machine computer myapp screen-size 1280x800     # hold it at this size, from 320x200 to 5120x2880
boxd machine computer myapp screen-size              # show the current size
boxd machine computer myapp screen-size --release    # it follows the viewer again

In the SDKs these are set_screen_size and release_screen_size (setScreenSize and releaseScreenSize), and every result says whether the screen is held. While the screen is held, someone in the Desktop view sees it scaled to fit their window, with its edge marked. They can still click and type, and nothing they do resizes it. A Held at 1280x800 · Release button at the top right lets them take it back. A hold survives standby and hibernation. It ends when it is released or the desktop restarts, so a stop or a reboot ends it too.

Claude and OpenAI agents

The SDKs have adapters that carry out a model’s computer-use tool calls on a machine: ClaudeComputer for Claude’s computer use tool and OpenAIComputer for OpenAI’s computer tool. They need neither model’s package, since they take and return plain data, and they work with the official SDKs.

import anthropic
from boxd import Boxd
from boxd.computer_use import ClaudeComputer

boxd = Boxd()
claude = anthropic.Anthropic()
computer = ClaudeComputer(boxd, "my-machine")
messages = [{"role": "user", "content": "Open example.com and read me its heading"}]
for _ in range(30):
    response = claude.messages.create(
        model="claude-opus-5-5",
        max_tokens=16000,
        tools=[computer.tool],
        messages=messages,
    )
    messages.append({"role": "assistant", "content": response.content})
    results = computer.run(response.content)     # one tool_result per action
    if not results:                              # no actions: Claude is done
        break
    messages.append({"role": "user", "content": results})
computer.release()

import Anthropic from "@anthropic-ai/sdk";
import { Boxd, ClaudeComputer } from "@boxd-sh/sdk";

const boxd = new Boxd();
const anthropic = new Anthropic();
const computer = new ClaudeComputer(boxd, "my-machine");

const messages: Anthropic.MessageParam[] = [
  { role: "user", content: "Open example.com and read me its heading" },
];
for (let turn = 0; turn < 30; turn++) {
  const response = await anthropic.messages.create({
    model: "claude-opus-5-5",
    max_tokens: 16000,
    tools: [computer.tool],
    messages,
  });
  messages.push({ role: "assistant", content: response.content });
  const results = await computer.run(response.content); // one tool_result per action
  if (results.length === 0) break;                       // no actions: Claude is done
  messages.push({ role: "user", content: results });
}
await computer.release();

OpenAIComputer works the same way: pass computer.tool in tools, and computer.run(response.output) returns a computer_call_output for each computer_call to send with your next request. What the adapters do for you:

Scale. The model sees screenshots scaled down to fit, 1280x800 for Claude and 1600x900 for OpenAI, and its coordinates are scaled back to the screen. max_width and max_height (maxWidth and maxHeight) change the size.
Hold the screen. The first action holds it at 1280x800 until you call release. screen_size= (screenSize) picks another size, and None (null) leaves the screen alone.
Stop at a failure. run carries out a turn’s actions in order and stops at the first that fails. The model gets that error, and the actions after it are reported as not run.
Leave safety checks to you. A computer_call from OpenAI with pending safety checks raises PendingSafetyCheckError without running anything, unless acknowledge_safety_checks (acknowledgeSafetyChecks) on run accepts them.

In Python, AsyncClaudeComputer and AsyncOpenAIComputer take an AsyncBoxd.

The model does whatever it decides to on the machine, as the machine’s user. Give it a machine of its own with nothing on it you’d mind losing. An isolated machine is cut off from your other machines and your credentials, and an egress allowlist limits what it can reach. Computer use works on an isolated machine from outside it, through the laptop CLI, the SDKs or the API.

A machine in another region

Name the region as you would for any machine command. See Regions.

boxd machine computer web.nyc cursor
boxd machine computer web --region nyc click 640 400

The SDK calls take region= ({ region }), and so do the adapters. On screenshot, region is the part of the screen to capture, so name the machine’s region with c.in_region("nyc").screenshot(id) (computer.inRegion("nyc").screenshot(id)).

When an action fails

A request that doesn’t fit, such as a point off the screen or a key name boxd doesn’t know, is refused before anything happens, with a message saying why. In the SDKs it raises an error with the code invalid_argument.
Any other failure may have happened after the action ran. The SDKs send a mouse or keyboard action again only when it can’t have reached the machine, so after a failure take a screenshot to see where things stand.

Limits


Held screen size	320x200 to 5120x2880
`type`	10,000 characters per call
`key`	1,000 characters of keys and 1,000 presses per call, `--repeat` up to 100, `--hold-ms` up to 5 minutes (not with `--repeat`)
`scroll`	100 wheel clicks each way
`drag`	100 points
`wait`	5 minutes
`--delay` before a screenshot	1 minute

A full-size PNG of a large screen can pass 30 MB. Ask for less with --max-width, --max-height or JPEG when you don’t need every pixel.

Getting started

Guides

Integrations and automations

Use cases

References

Computer use

Try it

How it works

From the CLI

From the SDKs

Hold the screen

Claude and OpenAI agents

A machine in another region

When an action fails

Limits

​Try it

​How it works

​From the CLI

​From the SDKs

​Hold the screen

​Claude and OpenAI agents

​A machine in another region

​When an action fails

​Limits

Try it

How it works

From the CLI

From the SDKs

Hold the screen

Claude and OpenAI agents

A machine in another region

When an action fails

Limits