> ## Documentation Index
> Fetch the complete documentation index at: https://docs.boxd.sh/llms.txt
> Use this file to discover all available pages before exploring further.

# Computer use

> Let an agent see and control a machine's desktop with screenshots, the mouse and the keyboard, from the CLI, the SDKs or the API.

Computer use gives an agent or a script a machine's [desktop](https://docs.boxd.sh/guides/desktop.md) the way a person at the screen has it: take a screenshot, decide what to do, then click, drag, scroll and type. It is the same desktop the **Desktop** view shows, so you can watch every step and step in.

Use it for GUI apps that aren't a web page, and for pages a browser tool can't work with. For ordinary web pages, [`agent-browser`](https://docs.boxd.sh/guides/launch-a-browser) is the better tool, because it reads and acts on the page itself rather than on pixels.

## Try it

```bash theme={"theme":"github-dark"}
boxd machine new demo
boxd machine exec demo -- agent-browser open https://example.com   # something to look at
boxd machine computer demo screenshot -o screen.png
boxd machine computer demo click 640 400 --screenshot after.png
```

Open `boxd machine desktop demo` to watch it happen.

## How it works

* **One action per call.** Every call answers with the screen's size and where the pointer is. Any action can also take a screenshot after itself in the same call, so a look-act-look loop costs one call per step.
* **Coordinates are screen pixels** from the top left. A screenshot scaled down with `--max-width` or `--max-height` is smaller than the screen, so scale coordinates you read off it back up before clicking. The [agent adapters](#claude-and-openai-agents) do this for you.
* **The desktop starts on demand.** The first action starts it if it isn't running, which can take about 10 seconds on a machine that has just started. Later actions don't wait for it.
* **Sleeping machines wake.** A machine on standby or hibernated is woken by an action. A stopped machine isn't started for you, so the call fails until you start it.
* **Who can use it.** Anyone who can run commands on a machine: its owner, any member of the organization when the machine is shared, and API keys with access to it. Actions run as the machine's user.

Computer use needs the desktop, so the [note about machine images](https://docs.boxd.sh/guides/desktop.md#open-it) on the Desktop page applies here too.

## From the CLI

The laptop CLI and the CLI inside a machine have the same commands. Inside a machine, leave out the name to act on that machine, or name another machine you can reach.

```bash theme={"theme":"github-dark"}
boxd machine computer myapp screenshot -o screen.png     # --format jpeg|webp, --quality, --max-width, --max-height, --crop X,Y,W,H, --delay
boxd machine computer myapp click 640 400                # --button right, --double, --triple, --count N, --modifiers ctrl+shift
boxd machine computer myapp move 100 100
boxd machine computer myapp mouse-down 100 100           # a drag step by step; also mouse-up
boxd machine computer myapp drag 100 100 400 300         # through more points: X Y X Y ...
boxd machine computer myapp scroll 640 400 --down 5      # wheel clicks; --up, --left, --right
boxd machine computer myapp type "hello world"           # `-` reads the text from stdin
boxd machine computer myapp key ctrl+l                   # Return, alt+Tab, ctrl+shift+t; --repeat N, --hold-ms MS
boxd machine computer myapp wait 1000                    # milliseconds
boxd machine computer myapp cursor --json                # {"screen": {...}, "cursor": {...}}
```

Every action takes `--screenshot FILE` and the same image options as `screenshot`:

```bash theme={"theme":"github-dark"}
boxd machine computer myapp click 640 400 --screenshot after.png --delay 500
boxd machine computer myapp key Return --screenshot - --format jpeg > after.jpg
```

The format defaults to the file's extension, then PNG, and `-` is stdout. Key names are X key names, as `xdotool` takes them, plus common aliases such as `enter`, `esc`, `cmd` and `pageup`.

> [!NOTE]
>
> `--crop` picks the part of the screen. `--region` is the machine's [region](https://docs.boxd.sh/guides/regions.md), as on every machine command.

## From the SDKs

`machines.computer` has the same actions. See the [Python](https://docs.boxd.sh/reference/python-sdk.md#computer-use) and [TypeScript](https://docs.boxd.sh/reference/typescript-sdk.md#computer-use) references for all of them.

```python Python theme={"theme":"github-dark"}
c = boxd.machines.computer

r = c.screenshot("myapp", max_width=1280)
r.screenshot.data                                  # PNG bytes
c.click("myapp", 640, 400)
c.type("myapp", "hello")
r = c.key("myapp", "Return", screenshot=True)      # and a screenshot after it, same call
r.screen_width, r.screen_height, r.cursor
```

```typescript TypeScript theme={"theme":"github-dark"}
const computer = boxd.machines.computer;

const { screenshot } = await computer.screenshot("myapp", { maxWidth: 1280 });
screenshot.data;                                   // PNG bytes
await computer.click("myapp", { x: 640, y: 400 });
await computer.type("myapp", "hello");
const r = await computer.key("myapp", "Return", { screenshot: true });  // and a screenshot after it, same call
r.screenWidth, r.screenHeight, r.cursor;
```

From any other language, call `ComputerAction` on the [gRPC API](https://docs.boxd.sh/reference/grpc-api.md#computer-use).

## Hold the screen

By default the screen follows whoever is watching it: a desktop nobody has opened is 1280x800, and while someone has the Desktop view open, the screen is the size of their window. That can change the screen's size in the middle of an agent's work, and coordinates it worked out a moment ago then point at the wrong place.

An agent should hold the screen while it works:

```bash theme={"theme":"github-dark"}
boxd machine computer myapp screen-size 1280x800     # hold it at this size, from 320x200 to 5120x2880
boxd machine computer myapp screen-size              # show the current size
boxd machine computer myapp screen-size --release    # it follows the viewer again
```

In the SDKs these are `set_screen_size` and `release_screen_size` (`setScreenSize` and `releaseScreenSize`), and every result says whether the screen is held.

While the screen is held, someone in the Desktop view sees it scaled to fit their window, with its edge marked. They can still click and type, and nothing they do resizes it. A **Held at 1280x800 · Release** button at the top right lets them take it back.

A hold survives standby and hibernation. It ends when it is released or the desktop restarts, so a stop or a reboot ends it too.

## Claude and OpenAI agents

The SDKs have adapters that carry out a model's computer-use tool calls on a machine: `ClaudeComputer` for [Claude's computer use tool](https://platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool) and `OpenAIComputer` for OpenAI's computer tool. They need neither model's package, since they take and return plain data, and they work with the official SDKs.

```python Python theme={"theme":"github-dark"}
import anthropic
from boxd import Boxd
from boxd.computer_use import ClaudeComputer

boxd = Boxd()
claude = anthropic.Anthropic()
computer = ClaudeComputer(boxd, "my-machine")
messages = [{"role": "user", "content": "Open example.com and read me its heading"}]
for _ in range(30):
    response = claude.messages.create(
        model="claude-opus-5-5",
        max_tokens=16000,
        tools=[computer.tool],
        messages=messages,
    )
    messages.append({"role": "assistant", "content": response.content})
    results = computer.run(response.content)     # one tool_result per action
    if not results:                              # no actions: Claude is done
        break
    messages.append({"role": "user", "content": results})
computer.release()
```

```typescript TypeScript theme={"theme":"github-dark"}
import Anthropic from "@anthropic-ai/sdk";
import { Boxd, ClaudeComputer } from "@boxd-sh/sdk";

const boxd = new Boxd();
const anthropic = new Anthropic();
const computer = new ClaudeComputer(boxd, "my-machine");

const messages: Anthropic.MessageParam[] = [
  { role: "user", content: "Open example.com and read me its heading" },
];
for (let turn = 0; turn < 30; turn++) {
  const response = await anthropic.messages.create({
    model: "claude-opus-5-5",
    max_tokens: 16000,
    tools: [computer.tool],
    messages,
  });
  messages.push({ role: "assistant", content: response.content });
  const results = await computer.run(response.content); // one tool_result per action
  if (results.length === 0) break;                       // no actions: Claude is done
  messages.push({ role: "user", content: results });
}
await computer.release();
```

`OpenAIComputer` works the same way: pass `computer.tool` in `tools`, and `computer.run(response.output)` returns a `computer_call_output` for each `computer_call` to send with your next request.

What the adapters do for you:

* **Scale.** The model sees screenshots scaled down to fit, 1280x800 for Claude and 1600x900 for OpenAI, and its coordinates are scaled back to the screen. `max_width` and `max_height` (`maxWidth` and `maxHeight`) change the size.
* **Hold the screen.** The first action holds it at 1280x800 until you call `release`. `screen_size=` (`screenSize`) picks another size, and `None` (`null`) leaves the screen alone.
* **Stop at a failure.** `run` carries out a turn's actions in order and stops at the first that fails. The model gets that error, and the actions after it are reported as not run.
* **Leave safety checks to you.** A `computer_call` from OpenAI with pending safety checks raises `PendingSafetyCheckError` without running anything, unless `acknowledge_safety_checks` (`acknowledgeSafetyChecks`) on `run` accepts them.

In Python, `AsyncClaudeComputer` and `AsyncOpenAIComputer` take an `AsyncBoxd`.

> [!WARNING]
>
> The model does whatever it decides to on the machine, as the machine's user. Give it a machine of its own with nothing on it you'd mind losing. An [isolated machine](https://docs.boxd.sh/use-cases/sandboxes.md) is cut off from your other machines and your credentials, and an [egress allowlist](https://docs.boxd.sh/guides/egress-allowlist.md) limits what it can reach. Computer use works on an isolated machine from outside it, through the laptop CLI, the SDKs or the API.

## A machine in another region

Name the region as you would for any machine command. See [Regions](https://docs.boxd.sh/guides/regions.md).

```bash theme={"theme":"github-dark"}
boxd machine computer web.nyc cursor
boxd machine computer web --region nyc click 640 400
```

The SDK calls take `region=` (`{ region }`), and so do the adapters. On `screenshot`, `region` is the part of the screen to capture, so name the machine's region with `c.in_region("nyc").screenshot(id)` (`computer.inRegion("nyc").screenshot(id)`).

## When an action fails

* **A request that doesn't fit**, such as a point off the screen or a key name boxd doesn't know, is refused before anything happens, with a message saying why. In the SDKs it raises an error with the code `invalid_argument`.
* **Any other failure** may have happened after the action ran. The SDKs send a mouse or keyboard action again only when it can't have reached the machine, so after a failure take a screenshot to see where things stand.

## Limits

| | |
| - | - |
| Held screen size | 320x200 to 5120x2880 |
| `type` | 10,000 characters per call |
| `key` | 1,000 characters of keys and 1,000 presses per call, `--repeat` up to 100, `--hold-ms` up to 5 minutes (not with `--repeat`) |
| `scroll` | 100 wheel clicks each way |
| `drag` | 100 points |
| `wait` | 5 minutes |
| `--delay` before a screenshot | 1 minute |

A full-size PNG of a large screen can pass 30 MB. Ask for less with `--max-width`, `--max-height` or JPEG when you don't need every pixel.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.