How to Control Mobile Devices with AI: Build a 15-Minute Automation Assistant

The Pain Points You've Probably Faced

When doing mobile development or writing automation scripts, you've likely experienced these frustrations:

1. Writing Appium Scripts: Setting up the environment takes half a day. You must maintain separate XCUITest and Espresso APIs. Every time the app UI changes, selectors break.
2. Data Scraping: Manually controlling simulators to tap, swipe, and recognize screenshots, or paying for commercial SDKs.
3. Daily Micro-Tasks: Something like "copy today's notification messages from this app" might only take a few taps, but the cost of writing code for it is too high.

What if you could just tell the AI what you want to do, and it automatically handles clicking, swiping, typing, and reading data on your phone? That would be a game-changer.

Today, we'll build exactly that. The whole process takes under 15 minutes. By the end, you'll be able to use AI assistants like Claude or Cursor to directly control your phone (or emulator) for automation and data scraping.

Prerequisites

Before starting, ensure you have the following ready:

• Node.js v20+: The project is distributed via npm. Install Node (using nvm is highly recommended).
• Mobile Development Environment (Choose One):
  • iOS Path: Mac + Xcode command line tools (xcode-select --install).
  • Android Path: Android SDK + adb accessible (enable Platform Tools in Android Studio's SDK Manager).
• An AI Client: Claude Desktop, Cursor, VS Code + Copilot, Codex, Gemini CLI, or any tool supporting the MCP protocol.

Why MCP? MCP (Model Context Protocol) is an open standard by Anthropic that allows AI models to securely call external tools. Here, we expose mobile control capabilities to the AI via an MCP Server, effectively giving AI "hands" to control your phone.

15-Minute Setup Tutorial

Step 1: Install & Configure the mobile-mcp Server

The project is published on npm. You can launch it with a single command—no repo cloning or manual compilation needed:

{
  "mcpServers": {
    "mobile-mcp": {
      "command": "npx",
      "args": ["-y", "@mobilenext/mobile-mcp@latest"]
    }
  }
}

Where to put this config? It depends on your AI client:

Recommendation: Claude Desktop or Cursor are highly recommended due to their mature MCP support. Once added, they work immediately without extra configuration.

Step 2: Start a Simulator or Connect a Device

The MCP Server doesn't create devices; you need to start them beforehand. For Android:

## List available emulators
emulator -list -avds

## Start an emulator
emulator @Pixel_6_API_33

## Verify device is online
adb devices

For iOS Simulator:

## List available simulators
xcrun simctl list devices

## Boot an iPhone 16 simulator
xcrun simctl boot "iPhone 16"

Key Concept: The MCP Server automatically discovers connected devices via adb (Android) or the XCUITest Accessibility Tree (iOS). Devices must be online first for the Server to work.

Step 3: Verify Connection

Restart your AI client (e.g., close and reopen Claude Desktop) and prompt:

List all available mobile devices

If configured correctly, you'll see:

Found 1 device:
- Pixel_6_API_33 (Android Emulator, booted)

If the list is empty, the emulator isn't running, or there's an issue with adb/Xcode toolchains. Fix device connectivity first.

Hands-On: Complete Tasks with Natural Language

Let's run a real-world scenario: Open a browser on the simulator, search for news, and copy back the title.

Scenario 1: Simple Browsing & Scraping

In Claude Desktop or Cursor (Agent mode), input:

Open the browser, go to https://news.ycombinator.com,
scrape the title and link of the first news item, and return them to me.

The AI will sequentially call these MCP tools (no code needed):

1. mobile_list_apps → Find Chrome/Browser
2. mobile_launch_app → Launch the browser
3. mobile_open_url → Open the URL
4. mobile_list_elements_on_screen → Read the page's accessibility tree
5. Extract title and link from the structured data.

This is MCP's value: You describe the goal, AI breaks it into a tool chain. No Appium scripts, no WebDriver connections, no HTML parsing. AI reads UI elements directly via the accessibility tree.

Scenario 2: Complex Automation (Great for QA)

Try this prompt:

Open the Calculator app on the device, calculate 123 + 456 = ?
Take a screenshot when the result appears, and save it.
Finally, tell me if the calculation result is 579.

AI will: Launch calculator → Tap numbers → Wait for result → Screenshot → Return structured result. You just verify.

Scenario 3: Multi-step Workflows

For full workflows, give the AI a complex instruction at once:

Open Instagram, search for "AI automation",
open the first profile, tap the Follow button,
then report back whether the operation succeeded.

Note: Multi-step operations rely on AI planning. Claude 3.5 Sonnet or GPT-4o work best. If AI gets stuck, manually guide it: "You are currently on the XX page, next you should tap the YY button."

Core MCP Tools Reference

You don't need to call these manually, but knowing them helps refine prompts:

Pro Tip: The project prioritizes the Accessibility Tree to understand UI, not screenshots + vision models. This means faster execution, zero image token costs, and structured data output. It only falls back to screenshots/coordinates when accessibility info is insufficient.

Troubleshooting & Pitfalls

1. Device List is Empty

Cause: Emulator not started, or adb lacks permissions.
Fix:

## Android
adb kill-server && adb start-server
adb devices  # Should show emulator-xxx

## iOS
xcrun simctl list devices | grep Booted  # Should show Booted devices

2. AI Says "mobile-mcp tool not found"

Fix: Restart the AI client. New MCP configs require a restart to load. For Claude Desktop, verify claude_desktop_config.json syntax (no trailing commas).

3. Inaccurate Taps

The MCP Server uses the Accessibility Tree by default. Some apps lack proper accessibility attributes. Prompt the AI to fall back to vision:

If you can't find the button's accessibility ID, take a screenshot and click the coordinates using visual recognition.

4. Real Device Setup is Complex?

iOS real devices require extra setup with go-ios and WebDriverAgent. It's highly recommended to test with simulators first. Refer to the Project Wiki for real-device guides.

5. Disable Telemetry

Anonymous telemetry is enabled by default. To disable:

MOBILEMCP_DISABLE_TELEMETRY=1 npx @mobilenext/mobile-mcp@latest

Or add an env field to your MCP config.

Summary & Next Steps

Today we went from zero to working:

1. ✅ Understood how MCP exposes mobile control to AI
2. ✅ Configured MCP Server and connected a simulator in under 15 minutes
3. ✅ Used natural language to browse, operate apps, and scrape data
4. ✅ Avoided common pitfalls like device discovery, tap targeting, and env setup

Where to go next:

• Turn repetitive daily tasks into prompt templates (e.g., "Every morning, scrape data from XX app and save to a file")
• Combine with mobilewright (Playwright for mobile) to convert AI-discovered flows into repeatable test scripts
• For large-scale testing or CI/CD, check out their Mobile Next Cloud service

With 6,244 stars, the open-source community is highly active. Submit Issues or join their Slack on the GitHub repo for tool-call help.

Drop a comment if you have questions. Happy automating!