Desktop Automation Guide

This guide provides practical, step-by-step instructions for implementing desktop automation with Claude Code. Follow along to build your first MCP server and automate UI workflows.

Core Concepts

Before building, it’s important to understand the key components:

Model-Context Protocol (MCP): A standardized protocol that allows AI models like Claude to interact with local tools and applications securely. The MCP Server you will build acts as a bridge between Claude and your desktop environment.
Desktop Automation: The process of using software to control a computer’s user interface (UI). In this context, we use Claude’s “computer use” capabilities to interpret natural language commands, see the screen, and control the mouse and keyboard, which then interacts with your MCP server to perform complex tasks.

Prerequisites

Before starting, ensure you have:

Claude Desktop installed
Node.js 18+ (for TypeScript examples) or Python 3.9+ (for Python examples)
An Anthropic API key (for computer use features)
Basic familiarity with terminal/command line

Part 1: Setting Up Your First MCP Server

Step 1: Create a Basic MCP Server (TypeScript)

Initialize your project:

mkdir my-mcp-server
cd my-mcp-server
npm init -y
npm install @modelcontextprotocol/sdk zod
npm install -D typescript @types/node tsx

Create src/index.ts:

import { Server } from '@modelcontextprotocol/sdk/server/index.js';
import { StdioServerTransport } from '@modelcontextprotocol/sdk/server/stdio.js';
import { z } from 'zod';
 
// Define tool schemas
const GreetingSchema = z.object({
  name: z.string().describe('Name to greet')
});
 
// Create server
const server = new Server({
  name: 'greeting-server',
  version: '1.0.0'
}, {
  capabilities: {
    tools: {}
  }
});
 
// Register tool
server.setRequestHandler('tools/list', async () => ({
  tools: [{
    name: 'greet',
    description: 'Generate a friendly greeting',
    inputSchema: {
      type: 'object',
      properties: {
        name: { type: 'string', description: 'Name to greet' }
      },
      required: ['name']
    }
  }]
}));
 
server.setRequestHandler('tools/call', async (request) => {
  if (request.params.name === 'greet') {
    const args = GreetingSchema.parse(request.params.arguments);
    return {
      content: [{
        type: 'text',
        text: `Hello, ${args.name}! Welcome to MCP development.`
      }]
    };
  }
  throw new Error(`Unknown tool: ${request.params.name}`);
});
 
// Start server
const transport = new StdioServerTransport();
server.connect(transport);

Add npm scripts to package.json:

{
  "scripts": {
    "build": "tsc",
    "start": "tsx src/index.ts"
  }
}

Step 2: Configure Claude Desktop

Find your Claude Desktop config file:
- macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
- Windows: %APPDATA%\Claude\claude_desktop_config.json
- Linux: ~/.config/Claude/claude_desktop_config.json

Add your MCP server:

{
  "mcpServers": {
    "greeting": {
      "command": "node",
      "args": ["/path/to/my-mcp-server/dist/index.js"],
      "env": {}
    }
  }
}

Restart Claude Desktop

Step 3: Test Your MCP Server

Open Claude Desktop
Type: “Use the greeting tool to say hello to Alice”
Claude should respond using your custom MCP server

Part 2: Building a File Operations Server

Step 1: Enhanced MCP Server with Security

import { Server } from '@modelcontextprotocol/sdk/server/index.js';
import { StdioServerTransport } from '@modelcontextprotocol/sdk/server/stdio.js';
import { z } from 'zod';
import * as fs from 'fs/promises';
import * as path from 'path';
 
// Configuration
const ALLOWED_DIRECTORIES = process.argv.slice(2);
 
// Schemas
const ReadFileSchema = z.object({
  path: z.string()
});
 
const WriteFileSchema = z.object({
  path: z.string(),
  content: z.string()
});
 
// Security: Path validation
async function validatePath(requestedPath: string): Promise<string> {
  const resolved = path.resolve(requestedPath);
  
  // Check if path is within allowed directories
  const isAllowed = ALLOWED_DIRECTORIES.some(dir => 
    resolved.startsWith(path.resolve(dir))
  );
  
  if (!isAllowed) {
    throw new Error(`Access denied: ${requestedPath}`);
  }
  
  return resolved;
}
 
// Create server with tools
const server = new Server({
  name: 'secure-file-server',
  version: '1.0.0'
}, {
  capabilities: {
    tools: {}
  }
});
 
// Register tools
server.setRequestHandler('tools/list', async () => ({
  tools: [
    {
      name: 'read_file',
      description: 'Read contents of a file',
      inputSchema: {
        type: 'object',
        properties: {
          path: { type: 'string', description: 'File path to read' }
        },
        required: ['path']
      }
    },
    {
      name: 'write_file',
      description: 'Write content to a file',
      inputSchema: {
        type: 'object',
        properties: {
          path: { type: 'string', description: 'File path to write' },
          content: { type: 'string', description: 'Content to write' }
        },
        required: ['path', 'content']
      }
    }
  ]
}));
 
// Handle tool calls
server.setRequestHandler('tools/call', async (request) => {
  try {
    switch (request.params.name) {
      case 'read_file': {
        const args = ReadFileSchema.parse(request.params.arguments);
        const validPath = await validatePath(args.path);
        const content = await fs.readFile(validPath, 'utf-8');
        
        return {
          content: [{
            type: 'text',
            text: content
          }]
        };
      }
      
      case 'write_file': {
        const args = WriteFileSchema.parse(request.params.arguments);
        const validPath = await validatePath(args.path);
        
        // Atomic write
        const tempPath = `${validPath}.tmp`;
        await fs.writeFile(tempPath, args.content, { flag: 'w' });
        await fs.rename(tempPath, validPath);
        
        return {
          content: [{
            type: 'text',
            text: `Successfully wrote to ${args.path}`
          }]
        };
      }
      
      default:
        throw new Error(`Unknown tool: ${request.params.name}`);
    }
  } catch (error) {
    return {
      content: [{
        type: 'text',
        text: `Error: ${error.message}`
      }],
      isError: true
    };
  }
});
 
// Start server
const transport = new StdioServerTransport();
server.connect(transport);

Step 2: Configure with Allowed Directories

Update Claude Desktop config:

{
  "mcpServers": {
    "secure-files": {
      "command": "node",
      "args": [
        "/path/to/secure-file-server/dist/index.js",
        "/Users/username/Documents",
        "/Users/username/Projects"
      ]
    }
  }
}

Part 3: Desktop Automation with Computer Use

Step 1: Set Up Computer Use Environment

Install dependencies:
```
pip install anthropic pillow
```

Create automation script:

import anthropic
from typing import Dict, Any
import base64
from PIL import ImageGrab
import io
 
class DesktopAutomation:
    def __init__(self, api_key: str):
        self.client = anthropic.Anthropic(api_key=api_key)
        
    def capture_screen(self) -> str:
        """Capture current screen and return base64 encoded image"""
        screenshot = ImageGrab.grab()
        buffer = io.BytesIO()
        screenshot.save(buffer, format='PNG')
        return base64.b64encode(buffer.getvalue()).decode()
    
    async def execute_task(self, task_description: str) -> Dict[str, Any]:
        """Execute a desktop automation task"""
        # Capture current screen state
        screen_base64 = self.capture_screen()
        
        # Send to Claude with computer use
        response = self.client.messages.create(
            model="claude-3.5-sonnet-20241022",
            max_tokens=1024,
            messages=[{
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": task_description
                    },
                    {
                        "type": "image",
                        "source": {
                            "type": "base64",
                            "media_type": "image/png",
                            "data": screen_base64
                        }
                    }
                ]
            }],
            tools=[{
                "type": "computer_use",
                "display_width": 1920,
                "display_height": 1080
            }]
        )
        
        return response

Step 2: Build a UI Testing Framework

class UITestFramework:
    def __init__(self, automation: DesktopAutomation):
        self.automation = automation
        self.test_results = []
    
    async def test_scenario(self, name: str, steps: list) -> bool:
        """Run a test scenario with natural language steps"""
        print(f"Running test: {name}")
        
        for i, step in enumerate(steps):
            try:
                result = await self.automation.execute_task(step)
                print(f"  Step {i+1}: ✓ {step}")
            except Exception as e:
                print(f"  Step {i+1}: ✗ {step}")
                print(f"    Error: {e}")
                self.test_results.append({
                    'test': name,
                    'step': i+1,
                    'description': step,
                    'status': 'failed',
                    'error': str(e)
                })
                return False
        
        self.test_results.append({
            'test': name,
            'status': 'passed'
        })
        return True
    
    async def run_test_suite(self, test_suite: Dict[str, list]):
        """Run multiple test scenarios"""
        for test_name, steps in test_suite.items():
            await self.test_scenario(test_name, steps)
        
        # Generate report
        passed = len([t for t in self.test_results if t.get('status') == 'passed'])
        total = len(test_suite)
        
        print(f"\nTest Results: {passed}/{total} passed")

Step 3: Example Test Suite

# Define your test suite
test_suite = {
    "User Login Flow": [
        "Click on the login button in the top right",
        "Enter 'testuser@example.com' in the email field",
        "Enter password in the password field",
        "Click the submit button",
        "Verify that the dashboard page is displayed"
    ],
    "Create New Document": [
        "Click on the 'New Document' button",
        "Type 'Test Document' as the title",
        "Click in the content area",
        "Type 'This is a test document created by automation'",
        "Click the save button",
        "Verify success message appears"
    ]
}
 
# Run tests
async def main():
    automation = DesktopAutomation(api_key="your-api-key")
    test_framework = UITestFramework(automation)
    await test_framework.run_test_suite(test_suite)
 
# Execute
import asyncio
asyncio.run(main())

Part 4: Building Desktop Extensions

Step 1: Create a Desktop Extension

Structure your extension:

my-extension/
├── manifest.json
├── server.js
├── package.json
└── README.md

Create manifest.json:

{
  "name": "My Desktop Extension",
  "version": "1.0.0",
  "description": "Custom MCP server for enterprise tools",
  "main": "server.js",
  "mcp": {
    "command": "node",
    "args": ["server.js"],
    "env": {
      "NODE_ENV": "production"
    }
  }
}

Bundle as .dxt file:

# Install bundler
npm install -g @anthropic/dxt-bundler
 
# Create extension
dxt-bundle my-extension/

Step 2: Submit to Directory

Complete the desktop extensions interest form
Include:
- Extension description
- Use cases
- Security considerations
- Testing results

Best Practices Summary

Security

Always validate inputs - Never trust user-provided paths or data
Implement rate limiting - Prevent resource exhaustion
Use atomic operations - Ensure data integrity
Audit all actions - Log tool invocations for security reviews
Build for Resilience - Implement patterns for error recovery and fault tolerance. See our comprehensive Resilience Patterns Guide for strategies like circuit breakers and intelligent retries.

Performance

Cache when possible - Reduce redundant operations
Use streaming for large data - Don’t load everything into memory
Implement timeouts - Prevent hanging operations
Monitor resource usage - Track CPU and memory consumption

User Experience

Provide clear error messages - Help users understand issues
Show progress for long operations - Keep users informed
Implement graceful degradation - Fall back when features unavailable
Document thoroughly - Include examples and troubleshooting

Troubleshooting

Common Issues

MCP server not appearing in Claude:
- Check config file syntax (valid JSON)
- Verify file paths are absolute
- Restart Claude Desktop
- Check server logs for errors
Permission denied errors:
- Ensure allowed directories are configured
- Check file system permissions
- Verify path resolution logic
Computer use not working:
- Confirm API key has computer use access
- Check screen resolution settings
- Verify coordinate calculations

Next Steps

Explore the Desktop Automation Deep Dive for advanced patterns
Review MCP Automation Scenarios for real-world examples
Check out the Testing Documentation for more testing strategies
Join the Claude MCP Community for support

This guide is continuously updated with new patterns and best practices.

Claude Code Research

Explorer

Desktop Automation Guide

Desktop Automation Guide

Core Concepts

Prerequisites

Part 1: Setting Up Your First MCP Server

Step 1: Create a Basic MCP Server (TypeScript)

Step 2: Configure Claude Desktop

Step 3: Test Your MCP Server

Part 2: Building a File Operations Server

Step 1: Enhanced MCP Server with Security

Step 2: Configure with Allowed Directories

Part 3: Desktop Automation with Computer Use

Step 1: Set Up Computer Use Environment

Step 2: Build a UI Testing Framework

Step 3: Example Test Suite

Part 4: Building Desktop Extensions

Step 1: Create a Desktop Extension

Step 2: Submit to Directory

Best Practices Summary

Security

Performance

User Experience

Troubleshooting

Common Issues

Next Steps

Graph View

Table of Contents