Overview

phone-eye

Let your AI agent see and operate a real Android phone — vision + UI-tree fusion over adb, for any MCP client

README / EN

Package documentation

Phone-Eye 📱👁️

Your AI agent can finally see and operate a real Android phone.

phone_look("what's on screen, where is the login button?")
  → "Login button at (540, 1830) — a green 'Sign in' …"
phone_tap(540, 1830)
phone_look("did the next page load?")

Works with any MCP client — Claude Code, Codex, Cursor, dsh, and friends.

What you need (plain words)

You provide	One-time or every time?	How hard?
An Android phone with USB debugging on	one-time, ~60 seconds (tap "Build number" 7× → enable USB debugging)	easy, step-by-step below
Plug the USB cable once	one-time — after that the tool switches the phone to Wi-Fi and you never need the cable again (unless the phone factory-resets)	trivial
A computer with Python 3.10+ and git (or just Docker — see step 3)	—	n/a
A vision model	one-time setup — bring any one of these: • an OpenAI API key (or any OpenAI-compatible: GLM, DeepSeek, local llama.cpp/Ollama…) • an existing MCP vision server	one env var, most people already have a key

That's everything. No app to install on the phone. No root. No extra server.

What it can and can't do (honest table)

✅ Stable	⚠️ Works but with caveats	❌ Not possible (any tool, not just us)
See the screen (screenshot + read it)	Locked screens can be read but not operated	Fully control a brand-new phone before you enable USB debugging yourself
Tap / swipe / type	Typing is ASCII (Chinese input needs a clipboard trick — known adb limit)	The very first "allow USB debugging?" popup on a new computer key — that one tap is yours
Survive Wi-Fi adb drops (auto-reconnect)	Some vendor ROMs restrict input on lock screens (e.g. MIUI)	iOS — different universe
Run 24/7 unattended; unexpected popups get read & handled by your agent	Vision quality depends on the model you bring
Multiple phones (one phone-eye process per phone, set `ANDROID_SERIAL` for each)

The 60-second rule: every Android requires one human moment — enable debugging + authorize once. After that, the phone belongs to your agent, even over Wi-Fi, even after reboots of the computer.

Setup (3 steps)

1. Enable USB debugging (once per phone)

Settings → About phone → tap Build number 7× (unlocks Developer options) → Developer options → USB debugging ON. (Got stuck? Tell us your phone model in Discussions — we'll walk you through it. MIUI/HyperOS may also ask you to sign into a Xiaomi account first.)

2. Install adb (if you don't have it), plug USB once, then go wireless

# macOS: brew install android-platform-tools · Ubuntu/Debian: sudo apt install adb
# Windows: scoop install adb  (or download Android platform-tools)
adb devices          # phone shows up? tap "Allow" on its popup — check "always allow"
adb shell ip route   # ← note the phone's Wi-Fi IP (e.g. 192.168.1.23) while still plugged
adb tcpip 5555       # switch to Wi-Fi mode (adb restarts; the USB entry disappears — normal)
adb connect 192.168.1.23:5555   # use the IP from above; then unplug the cable, forever

Phone IP alternatives if ip route prints nothing

Settings → Wi-Fi → your network → details shows the IP; or adb shell ip addr show wlan0 | grep inet.

3. Start phone-eye

git clone https://github.com/boheastill/phone-eye && cd phone-eye
pip install -r requirements.txt

# your eyes — pick ONE:
export PHONE_EYE_VISION_API_KEY=<key>                    # OpenAI / GLM / any compatible
#   (optional: PHONE_EYE_VISION_BASE_URL, PHONE_EYE_VISION_MODEL)
#   local & offline: ..._API_KEY=sk-noauth ..._BASE_URL=http://<host>:8080/v1 ..._MODEL=<your qwen-vl>

python server.py       # stdio MCP server — wire into your client:

Wire it into your client — pick yours:

# Claude Code (easiest):
claude mcp add phone-eye -- python /path/to/phone-eye/server.py
# Codex:
codex mcp add phone-eye --url stdio://python /path/to/phone-eye/server.py  # or see codex docs

// any MCP client (generic stdio shape):
{ "mcpServers": { "phone-eye": { "command": "python", "args": ["/path/to/phone-eye/server.py"] } } }

Docker (optional — no Python needed on the host)

The repo ships a Dockerfile (Python 3.12 + adb):

podman build -t phone-eye .        # or: docker build -t phone-eye .
# smoke: a JSON-RPC initialize reply on stdout means it boots:
printf '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","capabilities":{},"clientInfo":{"name":"smoke","version":"0"}}}\n' \
  | podman run --rm -i phone-eye

Use --network host so adb reaches a Wi-Fi phone and your vision endpoint:

"phone-eye": { "command": "podman", "args": ["run","--rm","-i","--network","host",
  "-e","ANDROID_SERIAL=192.168.1.23:5555","-e","PHONE_EYE_VISION_API_KEY=<key>","phone-eye"] }

Recommended vision models (any vision-capable chat model works): gpt-4o-mini (default), GLM glm-4.6v-flash (cheap), or run a local Qwen-VL via llama.cpp for fully-offline — screenshots then never leave your LAN.

What happens when something breaks

"No Android device reachable" → the tool already tried reconnecting; run adb connect <ip>:5555, or replug USB.
"No vision server reachable" → you haven't set a key; the error message tells you the exact two fixes.
Phone rebooted → Wi-Fi adb survives phone reboots on most ROMs; if not, one adb connect again.
Still stuck? Open a discussion — we answer, and we'll debug your setup with you. Bug reports and "it works on my X" notes are equally welcome.

Tools

Tool	What it does
`phone_look(question?)`	Ask a vision model about the live screen; fuses a UI-tree dump for exact text/button bounds
`phone_tap(x, y)`	Tap
`phone_swipe(x1, y1, x2, y2, ms?)`	Swipe
`phone_type(text)`	Type ASCII text
`phone_key(key)`	Press a hardware key — `wake` revives a sleeping phone (the unattended essential), back/home/recents navigate
`phone_intent(action, uri?, component?)`	Open any screen by Android intent (deep settings pages, app pages) without coordinates
`phone_screenshot()`	Save screenshot to disk, return path

Examples

Mobile web QA loop — the agent verifies its own work on a real screen
Surviving an OEM setup wizard — vision handles whatever pops up
Form regression check
Unattended sentinel — your agent on night watch: wake → unlock → intent → look → screenshot

Curious how it works — or want to modify it? Read the whitepaper (architecture, failure-classification decision tree, security model, how to add a verb). Running it as an always-on HTTP service behind your own fleet? docs/fleet.md.

Why "see", isn't this just adb?

UI-tree-only tools are blind to game canvases, images, and anything the accessibility tree can't show. Vision-only tools drift on coordinates. phone_look fuses both: the model answers what is this, the UI tree supplies exactly where. The day we dogfooded it, it discovered a USB-debugging popup on its own screen, read the buttons, and tapped "Allow" by itself — the story.

License

MIT. Verified on Redmi K40 Gaming / Android 13 — add your device to the table via PR.

Author: Bohea — independent industrial software engineer (Shenzhen). Operator HMIs · device integration · machine data into your customer's ERP. More runnable demos & engineering notes on the site.

phone-eye

phone-eye

Package documentation

Phone-Eye 📱👁️

What you need (plain words)

What it can and can't do (honest table)

Setup (3 steps)

1. Enable USB debugging (once per phone)

2. Install adb (if you don't have it), plug USB once, then go wireless

3. Start phone-eye

Docker (optional — no Python needed on the host)

What happens when something breaks

Tools

Examples

Why "see", isn't this just adb?

License

Related by capability