Anthropic Computer Use API: Desktop Automation Guide

What Is Computer Use API?

Computer Use represents a paradigm shift in AI capabilities. Rather than building specialized tools for individual tasks, Anthropic is teaching Claude general computer skills—enabling it to use the same interfaces, applications, and workflows that humans use every day.

Released in public beta on October 22, 2024, Computer Use makes Claude Sonnet 3.5 the first frontier AI model to offer autonomous desktop control. Companies like Asana, Canva, Cognition, DoorDash, Replit, and The Browser Company are already exploring applications that require dozens or even hundreds of steps to complete.

Key Capabilities

• Visual Understanding: Analyze screenshots to understand UI elements, content, and context
• Precise Mouse Control: Move cursor and click with pixel-perfect accuracy
• Keyboard Input: Type text, use keyboard shortcuts, and navigate interfaces
• Multi-Step Workflows: Chain actions together to complete complex tasks
• Error Recovery: Adapt to unexpected UI changes and error conditions

Current API Version

As of January 2025, Computer Use requires the API header anthropic-beta: computer-use-2025-01-24 with the claude-sonnet-4-5 model. The API is actively evolving with regular updates to improve accuracy and reliability.

How Computer Use Works

Computer Use operates through a continuous feedback loop where Claude analyzes the current screen state, decides on actions, and observes the results—similar to how a human user interacts with a computer.

The Execution Cycle

Why Pixel Counting Matters

When you ask Claude to click a button, it needs to translate "the blue submit button in the lower right" into exact pixel coordinates like (1245, 867). Traditional computer vision approaches struggle with this translation across different screen sizes and layouts.

Anthropic's solution was to train Claude to count pixels from reference points (screen edges, known UI elements) to target locations. This skill enables reliable cursor positioning regardless of screen resolution, DPI scaling, or application layout.

API Setup & Configuration

Setting up Computer Use requires the Anthropic SDK and proper configuration for desktop automation. The fastest way to get started is using Anthropic's official Docker container with a preconfigured environment.

Quick Start with Docker

The official anthropic-quickstarts/computer-use-demo repository provides a one-liner setup that spins up an Ubuntu 22.04 container with VNC server, desktop environment, and necessary tools:

docker run -d \
  --name claude-computer-use \
  -p 5900:5900 \
  -e ANTHROPIC_API_KEY=your_api_key_here \
  anthropic/computer-use-demo

Connect to the VNC server on localhost:5900 to view Claude's actions in real-time. This container includes:

• Ubuntu 22.04 with XFCE desktop environment
• Firefox browser for web automation
• PyAutoGUI for mouse/keyboard control
• Python environment with Anthropic SDK

Python SDK Installation

For custom implementations, install the required libraries:

pip install anthropic pyautogui pillow

# For screenshot capture
pip install mss

# For image processing
pip install opencv-python numpy

Basic API Configuration

import anthropic
import pyautogui
from PIL import Image
import io
import base64

# Initialize Anthropic client
client = anthropic.Anthropic(
    api_key="your_api_key_here"
)

# Configure Computer Use beta header
COMPUTER_USE_BETA = "computer-use-2025-01-24"

# Define available tools
tools = [
    {
        "type": "computer_20250124",
        "name": "computer",
        "display_width_px": 1920,
        "display_height_px": 1080,
        "display_number": 1
    },
    {
        "type": "text_editor_20250124",
        "name": "str_replace_editor"
    },
    {
        "type": "bash_20250124",
        "name": "bash"
    }
]

API Pricing & Limits

Computer Use runs on Claude Sonnet 4.5 with standard API pricing of $3 per million input tokens and $15 per million output tokens. Additional considerations:

• System Prompt: 466 tokens for automated tool selection
• Tool Definitions: 499 tokens per tool
• Screenshot Images: Typically 100-200 tokens per screenshot depending on resolution

Screenshot Analysis

Screenshot analysis is the foundation of Computer Use. Claude needs to understand what's currently visible on screen to decide which actions to take. Let's explore how to capture, encode, and send screenshots to the API.

Capturing Screenshots

import mss
import base64
from PIL import Image
from io import BytesIO

def capture_screenshot():
    """Capture screenshot and encode as base64"""
    with mss.mss() as sct:
        # Capture primary monitor
        monitor = sct.monitors[1]
        screenshot = sct.grab(monitor)

        # Convert to PIL Image
        img = Image.frombytes(
            'RGB',
            screenshot.size,
            screenshot.rgb
        )

        # Optimize size (reduce resolution if needed)
        # Computer Use works best with 1920x1080 or smaller
        max_width = 1920
        if img.width > max_width:
            ratio = max_width / img.width
            new_size = (max_width, int(img.height * ratio))
            img = img.resize(new_size, Image.Resampling.LANCZOS)

        # Convert to base64
        buffer = BytesIO()
        img.save(buffer, format='PNG', optimize=True)
        img_str = base64.b64encode(buffer.getvalue()).decode()

        return {
            'type': 'image',
            'source': {
                'type': 'base64',
                'media_type': 'image/png',
                'data': img_str
            }
        }

# Example usage
screenshot = capture_screenshot()

Sending Screenshots to Claude

def analyze_screen(task_description: str):
    """Send screenshot to Claude for analysis"""
    screenshot = capture_screenshot()

    response = client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=1024,
        tools=tools,
        messages=[
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": f"Task: {task_description}\n\nAnalyze this screenshot and determine the next action."
                    },
                    screenshot
                ]
            }
        ],
        betas=[COMPUTER_USE_BETA]
    )

    return response

# Example: Analyze a login screen
result = analyze_screen("Fill out the login form with username 'demo' and click submit")

Understanding Claude's Analysis

Claude's vision model analyzes screenshots to identify:

• UI Elements: Buttons, text fields, dropdowns, menus, checkboxes
• Text Content: Labels, instructions, error messages, form fields
• Visual Context: Application state, active windows, loaded pages
• Spatial Layout: Element positions, sizes, relationships

Mouse & Keyboard Control

The Computer tool provides mouse and keyboard functions that Claude invokes based on screenshot analysis. Let's explore how to implement and handle these control mechanisms.

Mouse Operations

Claude uses tool calls to control the mouse. Here's how to implement the handlers:

import pyautogui
import time

def handle_mouse_move(x: int, y: int):
    """Move cursor to specific coordinates"""
    pyautogui.moveTo(x, y, duration=0.2)
    time.sleep(0.1)
    return {"success": True, "action": f"moved to ({x}, {y})"}

def handle_left_click():
    """Perform left mouse click"""
    pyautogui.click()
    time.sleep(0.2)
    return {"success": True, "action": "left click"}

def handle_right_click():
    """Perform right mouse click"""
    pyautogui.rightClick()
    time.sleep(0.2)
    return {"success": True, "action": "right click"}

def handle_double_click():
    """Perform double click"""
    pyautogui.doubleClick()
    time.sleep(0.2)
    return {"success": True, "action": "double click"}

def handle_mouse_drag(start_x: int, start_y: int, end_x: int, end_y: int):
    """Drag from start to end coordinates"""
    pyautogui.moveTo(start_x, start_y)
    time.sleep(0.1)
    pyautogui.dragTo(end_x, end_y, duration=0.5)
    time.sleep(0.2)
    return {"success": True, "action": f"dragged from ({start_x}, {start_y}) to ({end_x}, {end_y})"}

Keyboard Operations

def handle_type_text(text: str):
    """Type text with natural typing speed"""
    pyautogui.write(text, interval=0.05)
    time.sleep(0.2)
    return {"success": True, "action": f"typed text: {text[:50]}..."}

def handle_key_press(key: str):
    """Press a single key or key combination"""
    pyautogui.press(key)
    time.sleep(0.1)
    return {"success": True, "action": f"pressed key: {key}"}

def handle_hotkey(*keys):
    """Press key combination (e.g., Ctrl+C)"""
    pyautogui.hotkey(*keys)
    time.sleep(0.2)
    return {"success": True, "action": f"hotkey: {'+'.join(keys)}"}

# Example keyboard shortcuts
def common_shortcuts():
    return {
        "copy": lambda: handle_hotkey('ctrl', 'c'),
        "paste": lambda: handle_hotkey('ctrl', 'v'),
        "save": lambda: handle_hotkey('ctrl', 's'),
        "undo": lambda: handle_hotkey('ctrl', 'z'),
        "select_all": lambda: handle_hotkey('ctrl', 'a'),
        "tab": lambda: handle_key_press('tab'),
        "enter": lambda: handle_key_press('enter'),
        "escape": lambda: handle_key_press('escape')
    }

Tool Call Handler

Process Claude's tool calls to execute the requested actions:

def process_tool_call(tool_use):
    """Execute tool calls from Claude's response"""
    tool_name = tool_use.name
    tool_input = tool_use.input

    # Computer tool actions
    if tool_name == "computer":
        action = tool_input.get("action")

        if action == "mouse_move":
            return handle_mouse_move(
                tool_input["coordinate"][0],
                tool_input["coordinate"][1]
            )
        elif action == "left_click":
            return handle_left_click()
        elif action == "right_click":
            return handle_right_click()
        elif action == "double_click":
            return handle_double_click()
        elif action == "type":
            return handle_type_text(tool_input["text"])
        elif action == "key":
            return handle_key_press(tool_input["text"])
        elif action == "screenshot":
            return capture_screenshot()

    # Text editor tool
    elif tool_name == "str_replace_editor":
        command = tool_input.get("command")
        if command == "view":
            with open(tool_input["path"], 'r') as f:
                return {"content": f.read()}
        elif command == "str_replace":
            # Implement file editing logic
            pass

    # Bash tool
    elif tool_name == "bash":
        import subprocess
        result = subprocess.run(
            tool_input["command"],
            shell=True,
            capture_output=True,
            text=True
        )
        return {
            "stdout": result.stdout,
            "stderr": result.stderr,
            "exit_code": result.returncode
        }

    return {"error": "Unknown tool or action"}

Workflow Automation Examples

Let's explore real-world automation workflows that demonstrate Computer Use capabilities and best practices.

def automate_form_filling(form_data: dict):
    """Fill web form with provided data"""

    # Initial prompt with form data
    task = f"""
    Fill out the registration form with this data:
    - First Name: {form_data['first_name']}
    - Last Name: {form_data['last_name']}
    - Email: {form_data['email']}
    - Phone: {form_data['phone']}

    Then click the Submit button.
    """

    messages = [{
        "role": "user",
        "content": [
            {"type": "text", "text": task},
            capture_screenshot()
        ]
    }]

    # Automation loop
    max_iterations = 20
    for i in range(max_iterations):
        response = client.messages.create(
            model="claude-sonnet-4-5",
            max_tokens=2048,
            tools=tools,
            messages=messages,
            betas=[COMPUTER_USE_BETA]
        )

        # Process tool calls
        if response.stop_reason == "tool_use":
            tool_results = []
            for content in response.content:
                if content.type == "tool_use":
                    result = process_tool_call(content)
                    tool_results.append({
                        "type": "tool_result",
                        "tool_use_id": content.id,
                        "content": str(result)
                    })

            # Add assistant response and tool results
            messages.append({"role": "assistant", "content": response.content})
            messages.append({"role": "user", "content": tool_results})

        # Check completion
        elif response.stop_reason == "end_turn":
            print("Form submission complete")
            break

    return True

import pandas as pd

def bulk_data_entry(csv_path: str, app_url: str):
    """Enter data from CSV into web application"""

    # Load data
    df = pd.read_csv(csv_path)

    # Open application
    task = f"Open web browser and navigate to {app_url}"
    automate_task(task)

    # Process each row
    for index, row in df.iterrows():
        print(f"Processing row {index + 1}/{len(df)}")

        task = f"""
        Enter the following data into the form:
        - Product: {row['product_name']}
        - Quantity: {row['quantity']}
        - Price: {row['price']}
        - Category: {row['category']}

        Click Save and wait for confirmation.
        Then click 'Add Another' to continue.
        """

        automate_task(task)

        # Brief pause between entries
        time.sleep(2)

    print("Data entry complete")

def test_checkout_flow():
    """Test e-commerce checkout process"""

    test_steps = [
        "Navigate to the product page",
        "Click 'Add to Cart' button",
        "Click the shopping cart icon",
        "Verify product appears in cart",
        "Click 'Proceed to Checkout'",
        "Fill shipping address with test data",
        "Select shipping method: Standard",
        "Enter payment info (test card)",
        "Click 'Place Order'",
        "Verify order confirmation appears"
    ]

    results = []

    for step in test_steps:
        print(f"Testing: {step}")

        task = f"""
        {step}

        After completing this action:
        1. Take a screenshot
        2. Verify the action succeeded
        3. Report any errors or unexpected behavior
        """

        result = automate_task(task)
        results.append({
            "step": step,
            "success": result.get("success", False),
            "notes": result.get("notes", "")
        })

        time.sleep(1)

    # Generate test report
    return generate_test_report(results)

def process_invoices(pdf_folder: str):
    """Extract data from invoices and enter into accounting system"""

    import os

    pdf_files = [f for f in os.listdir(pdf_folder) if f.endswith('.pdf')]

    for pdf_file in pdf_files:
        print(f"Processing {pdf_file}")

        # Open PDF and extract data
        task = f"""
        1. Open the PDF file at {os.path.join(pdf_folder, pdf_file)}
        2. Extract the following information:
           - Invoice number
           - Date
           - Vendor name
           - Total amount
           - Line items with descriptions and amounts
        3. Take note of all extracted data
        """

        extracted_data = automate_task(task)

        # Enter into accounting system
        entry_task = f"""
        1. Open the accounting software
        2. Click 'New Invoice Entry'
        3. Fill in the extracted data:
           {extracted_data}
        4. Attach the PDF file
        5. Click 'Save'
        6. Verify the entry was saved successfully
        """

        automate_task(entry_task)

        # Move processed file
        os.rename(
            os.path.join(pdf_folder, pdf_file),
            os.path.join(pdf_folder, 'processed', pdf_file)
        )

Safety Guidelines & Best Practices

Computer Use introduces new security considerations. Following Anthropic's safety guidelines is essential for responsible deployment.

Security Vulnerabilities

Anthropic acknowledges that Computer Use is susceptible to:

• Jailbreaking: Attempts to bypass safety guidelines through adversarial prompts
• Prompt Injection: Claude may follow commands found in on-screen content, potentially conflicting with user instructions
• Unintended Actions: Model errors could trigger destructive operations

Development Best Practices

def require_confirmation(action: str) -> bool:
    """Request human confirmation for sensitive actions"""
    print(f"\nClaude wants to perform: {action}")
    response = input("Allow this action? (yes/no): ")
    return response.lower() == "yes"

# In tool handler
if action_is_sensitive(action):
    if not require_confirmation(action):
        return {"error": "Action denied by user"}

import json
from datetime import datetime

def log_action(action_type: str, details: dict):
    """Log all Computer Use actions"""
    log_entry = {
        "timestamp": datetime.now().isoformat(),
        "action_type": action_type,
        "details": details
    }

    with open("computer_use_audit.jsonl", "a") as f:
        f.write(json.dumps(log_entry) + "\n")

# Log every tool call
log_action("mouse_click", {"x": 100, "y": 200})
log_action("type_text", {"text": "username"})

def automate_with_limits(task: str, max_steps: int = 50, timeout_seconds: int = 300):
    """Run automation with safety limits"""
    start_time = time.time()

    for step in range(max_steps):
        # Check timeout
        if time.time() - start_time > timeout_seconds:
            raise TimeoutError("Automation exceeded time limit")

        # Execute step
        result = execute_step(task)

        if result.get("complete"):
            return result

    raise RuntimeError("Exceeded maximum step count")

Anthropic's Safety Classifiers

Anthropic has developed new classifiers that identify when Computer Use is being employed and whether harmful actions are occurring. These classifiers help detect:

• Spam generation attempts
• Misinformation creation
• Fraud or malicious automation

Current Limitations

Computer Use is in public beta and has notable limitations. Understanding these constraints helps set realistic expectations and plan appropriate use cases.

Frequently Asked Questions