Claude is able to control computer screens

🤯

How computer use works

1. Provide Claude with computer use tools and a user prompt

• Add Anthropic-defined computer use tools to your API request.

• Include a user prompt that might require these tools, e.g., “Save a picture of a cat to my desktop.”

2. Claude decides to use a tool

• Claude loads the stored computer use tool definitions and assesses if any tools can help with the user’s query.

• If yes, Claude constructs a properly formatted tool use request.

• The API response has a stopreason of tooluse, signaling Claude’s intent.

3. Extract tool input, evaluate the tool on a computer, and return results

• On your end, extract the tool name and input from Claude’s request.

• Use the tool on a container or Virtual Machine.

• Continue the conversation with a new user message containing a toolresult content block.

4. Claude continues calling computer use tools until it's completed the task

• Claude analyzes the tool results to determine if more tool use is needed or the task has been completed.

• If Claude decides it needs another tool, it responds with another tooluse stopreason and you should return to step 3.

• Otherwise, it crafts a text response to the user.

Source:

https://docs.anthropic.com/en/docs/build-with-claude/computer-use