agents-io/PokeClaw

🌐 Landing Page — available in English · हिन्दी · 日本語 · Deutsch · 繁中

PokeClaw (PocketClaw) — On-Device AI Phone Agent

PokeClaw, also known as PocketClaw, is an open-source Android app for AI phone automation.

It can run Gemma 4 on-device for local, private phone control, and it also supports optional cloud models when you want stronger reasoning for harder tasks.

The current public build is a local-first prototype for turning an Android phone into an AI-operated device.

In Local mode, model execution stays inside your device. No account or API key is required for Local mode.

Everyone else:  Phone → Internet → Cloud API → Internet → Phone
                       💳Credit card needed, API key required. Monthly bill attached.
PokeClaw local: Phone → LLM → Phone
                       Local-first when you want it. Optional cloud when you need it.

AI can control your phone, with local-first execution and optional cloud help.

The current public build is open-source and already handles real chat, task, and automation flows on Android.

Monitor a WhatsApp contact and auto-reply:

Context-aware WhatsApp auto-reply:

https://github.com/user-attachments/assets/5a43d4d5-458a-4eea-a0a5-58d113255741

https://github.com/user-attachments/assets/5c2966c5-04e6-4b22-8d66-11915ae62096

☝️ Auto-reply demo: PokeClaw monitors messages from Mom, reads what she said, and replies based on context using the on-device LLM. Watch in higher resolution on YouTube

☝️ Context demo: Mom asks "what did I tell you to bring?" — the AI opens the chat, reads the full conversation on screen, sees the earlier message about wine, and replies correctly. This is the difference between context-aware and context-free replies.

https://github.com/user-attachments/assets/89999dd8-a1be-49ad-9419-60c2b38f6374

Why is the "hi" demo slow? That clip was recorded on a CPU-only Android device with no usable GPU or NPU path. Running Gemma 4 E2B on pure CPU takes about 45 seconds to warm up. On stronger phones it is much faster:

- Google Tensor G3/G4 (Pixel 8, Pixel 9)

- Snapdragon 8 Gen 2/3 (Galaxy S24, OnePlus 12)

- Dimensity 9200/9300 (recent MediaTek flagships)

- Snapdragon 7+ Gen 2+ (mid-range with GPU)

> On these devices, warmup drops to seconds. Same model, better hardware.

The Story

I'm building this solo. When Gemma 4 landed with native tool calling on LiteRT-LM, I wanted to know whether a phone could become a real on-device agent instead of just another chatbot. PokeClaw is the result.

The interesting part is not just chatting with a local model. The interesting part is getting a local model to read the screen, choose tools, operate apps, keep task state, and finish real phone workflows. That is exactly what this project is built for.

PokeClaw already supports fully on-device automation with Gemma 4 and optional cloud models for stronger task execution. The current focus is broader device support, more generic skills, more local model options, and a cleaner public release path.

If you hit something interesting, open an issue. Real device reports are how this gets better fast.

Product Direction

PokeClaw is not just a chat app with a few phone-control tricks glued on top.

At its core, it is becoming a mobile agent harness:

• a generic tool layer for phone control
• a task/runtime loop that lets a model choose and chain those tools
• playbooks, rules, and guards that can be iterated against real device QA
• a product shell on top so the same harness is usable by normal people, not just developers

That distinction matters. The long-term goal is not to hardcode one-off app flows forever. The goal is to build the strongest practical harness for AI phone agents on Android, then ship product experiences on top of that foundation.

That is also why the project invests so heavily in:

• generic tools before narrow workflows
• repeated real-device QA instead of one-off demos
• Cloud vs Local model comparisons on the same task families
• playbooks and rules only when the model proves it needs extra structure

That direction affects how bugs are prioritized. PokeClaw should fix deterministic harness, runtime, hardware, storage, accessibility, foreground-service, signing, and QA-runner problems before drilling into one flaky model task.

Examples of harness problems worth fixing immediately:

• the model cannot observe the relevant screen text
• a tool reports success when it actually failed
• task state leaks into the next run
• Accessibility, Notification Access, foreground service, or install/update state is wrong
• model download, storage, or LiteRT startup fails before the model can run
• GPU fallback is mislabeled, crashes, or hides the real fallback reason

Examples that should usually be treated as model-performance or exploratory-agent limits unless logs prove otherwise:

• one Cloud model fails one multi-step app flow once
• one Local model cannot reason through a complex UI path
• a workflow sometimes succeeds and sometimes fails because the model chooses a weak plan
• a prompt needs repeated attempts to pick the right cross-app path

Prompts, tools, skills, and playbooks should stay generic. Add structure when it improves a reusable class of tasks; avoid one-off prompt hacks or coordinate scripts just to make a single demo pass.

Deployment Positioning

There are already strong mobile-agent frameworks for developers, benchmarks, and cloud/desktop-controlled devices. DroidRun/Mobilerun, minitap/mobile-use, Mobile-Agent, and AppAgent-style systems are useful references, especially for planning, UI observation, benchmark design, and failure recovery. They are not the same deployment category as PokeClaw:

• DroidRun/Mobilerun local/open-source runs the agent loop in a Python CLI/container on a host machine and talks to a Portal APK through ADB/port forwarding. The Portal APK uses Android Accessibility for tree/gesture/screenshot capture, but it is not where the local LLM agent loop runs. Its phone-initiated path is cloud-backed.
• minitap/mobile-use, Mobile-Agent, AppAgent, and AppAgentX are external SDK/research harnesses that normally require Python plus ADB, UIAutomator, screenshots, Docker, cloud phones, or host-side model/API services.

PokeClaw should not try to become another PC/SDK/ADB-driven mobile automation framework. Its product lane is different:

• runs as an Android app on the user's own phone
• does not require a computer to be connected while tasks run
• uses Accessibility, Notification Access, foreground service, storage, and Android lifecycle handling inside the phone app
• runs the task loop inside the installed APK, with local LiteRT-LM or optional cloud LLMs behind the same harness
• exposes integration points for power users without making Tasker, MacroDroid, ADB, or a cloud phone fleet mandatory

External automation tools are still useful. Tasker, MacroDroid, Locale, and similar apps are good deterministic trigger engines. PokeClaw should let them trigger an AI task, but PokeClaw should remain the phone-resident execution harness.

See the UI

👉 Try the interactive demo on our landing page — click through every screen without installing anything.

What it does

The model picks the right tool, fills in the parameters, and executes. You don't configure anything per-app. It just reads the screen and acts.

Proven Quick Tasks

These are tasks we have already run end-to-end during on-device QA.

Local mode

• Summarize notifications
• Explain clipboard contents
• Analyze storage / apps and suggest cleanup targets
• Check whether the battery needs charging
• Report installed apps
• Report phone temperature
• Report Bluetooth state
• Report battery, storage, and Android version
• Run quick-task cards directly from the UI and return the result in chat
• Route contact-specific send / call tasks correctly and fail cleanly when the contact does not exist on the device

Cloud mode

• Send a WhatsApp message and auto-return to the same PokeClaw conversation
• Search inside YouTube in the real app
• Check what is trending on Twitter / X and summarize it
• Install or open Telegram from Play Store
• Open Reddit and search for pokeclaw
• Copy the latest email subject and Google it
• Draft an email saying you will be late
• Preserve task state and session history across cross-app execution and return

Benchmark & Real-Device QA

Every number below comes from repeated trials on a physical Pixel 8 Pro running release builds. No cherry-picked runs, no emulators. The full verified task list and tier breakdown is in thoughts/verified-task-capabilities.md.

Cloud (GPT-4.1) — 18/20 pass, real tasks on real phone

| Task | Result | Rounds | What happens | |---|---:|---:|---| | Search YouTube for lofi beats | ✅ | 9 | Opens YouTube, types query, hits search | | Open Chrome and search for weather | ✅ | 9 | Opens Chrome, types query, reads results | | Open Chrome and go to reddit.com | ✅ | 7 | URL navigation with node_id targeting | | Compose email to test@example.com | ✅ | 12 | Opens Gmail, fills To + Subject + Body | | Install Telegram from Play Store | ✅ | 14 | Search + tap Install + wait | | Turn on do not disturb | ✅ | 12 | Navigates Settings, toggles DND | | Open Settings, go to About Phone | ✅ | varies | Deep settings navigation | | Send hi to Mom on WhatsApp | ✅ | 5 | Opens WhatsApp, finds contact, types, sends | | Check my Instagram messages | ✅ | 5 | Handles typo ("instagarm"), opens correct app | | 部機仲有幾多storage | ✅ | 2 | Cantonese input, returns answer in 中文 | | 打開Instagram | ✅ | varies | Chinese command | | Draft an email saying I'll be late | ✅ 10/10 | 8 | Repeated trials: 100% pass rate | | Copy latest email subject and Google it | ✅ 8/10 | 15 | Gmail to Chrome cross-app flow, 80% pass rate |

All tasks use zero hardcoded app logic. The model reads the screen, picks tools, and figures out the flow on its own. Multi-language works out of the box, including Cantonese, Mandarin, and misspelled English.

Local (Gemma 4 E2B, fully on-device) — verified on CPU and GPU

| Task family | Result | CPU avg (Pixel 8 Pro) | GPU avg (Pixel 8 Pro) | Notes | |---|---:|---:|---:|---| | Clipboard explain | ✅ | 2m 43s | 2m 11s | Real clipboard read | | Notifications summary | ✅ | 2m 49s | 2m 53s | Reads live notifications and summarizes | | Battery advice | ✅ | 2m 12s | 2m 53s | Returns level + charging state | | Storage + apps cleanup advice | ✅ | 2m 33s | 3m 06s | Harness used to mislabel this as blocked because the answer mentioned the Contacts app |

For the current local-core quick-task bundle on a Pixel 8 Pro, Gemma 4 E2B passed 4/4 on both CPU and GPU. Cold-start average time was 2m 34s on CPU versus 2m 46s on GPU, so GPU is now verified and usable on this device, but it is not yet a cold-start speed win for this short task bundle. The value of the recent hardening work is stability and real backend verification, not inflated benchmark theater.

How it works

PokeClaw gives a small on-device LLM a set of tools (tap, swipe, type, open app, send message, enable auto-reply, etc.) and lets it decide what to do. The LLM sees a text representation of the current screen, picks an action, sees the result, picks the next action, until the task is done.

Local execution runs via LiteRT-LM with native tool calling. In Local mode, the model runs on-device.

Tools

The LLM has access to these tools and picks them autonomously:

| Tool | What it does | |------|-------------| | tap / swipe / long_press | Touch the screen | | input_text | Type into any text field | | open_app | Launch any installed app | | send_message | Full messaging flow: open app, find contact, type, send | | auto_reply | Monitor a contact and reply automatically using LLM | | get_screen_info | Read current UI tree | | take_screenshot | Capture screen | | finish | Signal task completion |

These tools are generic — they work with any app, any contact, any language. The LLM picks the right tool and fills in the parameters from your request.

Tools + Skills

Small on-device models get dramatically better when you give them a strong playbook. So we give PokeClaw reusable skills on top of generic tools.

The auto-reply feature is a good example. It doesn't work by magic — there's a predefined workflow behind it: open the chat → read all visible messages on screen → generate a context-aware reply → send it → go back to home. The model follows this recipe step by step. Every tool in that chain is generic: open_app works with any app, read_screen works on any screen, send_message works with any contact. The workflow just tells the model which tools to use and in what order.

This is what we're calling Skills — reusable workflows built from generic tools. We're actively designing a skill system inspired by Claude Code's skill architecture. The idea: anyone can write a skill as a simple text file that describes the steps, and the LLM follows it.

Some examples of what skills can do:

• Auto-reply: monitor notifications → open chat → read conversation → generate reply → send
• Morning briefing: open weather app → read temperature → open calendar → read today's events → open email → count unread → summarize everything
• Smart forward: catch a notification → open the message → read it → forward to another contact with a summary
• Auto-booking: open a booking app → search for a time slot → fill in details → confirm

Each skill is just a combination of the same generic tools (open_app, tap, type, read_screen, send_message, etc.) arranged in a specific order. The tools are the building blocks, the skills are the recipes.

Both are designed to be extensible. We're building the first 8-10 skills as built-in defaults. If the system works well, we'll open it up for the community to create and share their own tools and skills. You know your phone better than we do — you should be able to teach it new tricks.

As on-device models get smarter, more of this can become free-form. Right now, skills are how we get reliable automation out of a small local model while keeping the tool layer generic.

Download

Download APK

Note: If you are updating from an older public debug build and Android says the package is incompatible, uninstall the old build once and then install the latest APK fresh. Older public debug builds still receive the in-app update prompt, but they need a one-time reinstall before joining the stable-signed 0.6.x line.

Requirements

| | Minimum | Recommended | |---|---|---| | Android | 9+ | 12+ | | Architecture | arm64 | arm64 | | RAM | 8 GB | 12 GB+ | | Storage | 3 GB free (model download) | 5 GB+ | | GPU | Not required (CPU works) | Tensor G3/G4, Snapdragon 8 Gen 2+, Dimensity 9200+ | | Root | Not required | Not required |

⚠️ 8 GB gets you in the door. 12 GB+ is the sweet spot for the built-in Gemma 4 local models, especially if you want smoother multitasking and faster model bring-up.

Quick start

1. Install the APK 2. Grant Accessibility permission when prompted 3. If you want background monitor flows, also grant Notification Access 4. In Local mode, the model downloads on first local launch (~2.6 GB) 5. Switch to Chat or Task mode and start using it

Local mode needs no account and no API key. Cloud mode is optional.

Roadmap

This is the current direction for PokeClaw based on real device testing, open issues, and the most common feature requests.

Near-term

• Stabler public releases and upgrades. The release/signing path is being locked down so future public APKs upgrade cleanly instead of falling back to uninstall/reinstall behavior from the older debug-signed builds.
• External automation input. Tasker, MacroDroid, Locale, and similar Android automation tools are strong deterministic trigger engines. PokeClaw now exposes a production, user-enabled intent API so those tools can decide when to trigger and PokeClaw can handle the open-ended AI task execution. Next work is richer recipes, cancellation/status actions, and Tasker/MacroDroid callback-consumer QA.
• Persistent instructions and scoped rules. PokeClaw needs an inspectable instruction stack: hard platform/tool safety in code, user global instructions, app/channel-scoped rules, explicit user-approved memory, then the current task prompt. Screen, notification, and web content should remain untrusted context, not higher-priority instructions.
• Missed-call auto follow-up. A high-priority use case is: someone calls, you miss it, and PokeClaw automatically sends a follow-up message to that caller and keeps the status visible in the same chatroom. The preferred first path is SMS/API-first. WhatsApp follow-up is only worth adding if there is a reliable non-UI route; otherwise it should stay an explicit fallback path, not the core design.
• Lower-RAM local model options. Right now the built-in local model choices are still too heavy for a lot of mid-range phones. Smaller on-device models are high priority.
• More small local models for real device coverage. 1B–1.5B class models for lower-end phones are on the roadmap so more devices can at least run a usable local agent instead of being locked out by RAM limits.
• More reliable local model downloads. Resume/retry behavior, partial download cleanup, and corrupted-model detection are all being hardened so downloads survive weak connections and screen-off/resume cases better.
• Broader device compatibility. Samsung, Xiaomi, Dimensity, and low-RAM device issues are being used as real-world test cases for GPU→CPU fallback, model loading, accessibility reconnects, and generic UI control.
• More generic phone-control skills. We are continuing to replace brittle, app-specific assumptions with generic tools and reusable skills so tasks survive OEM UI changes better.

In progress

• Import your own local .litertlm models. User-accessible local model import is on the roadmap so you can bring your own LiteRT model instead of being locked to the built-in download list.
• Custom local model sources. We want PokeClaw to go beyond a fixed built-in catalog and support user-defined model sources, including direct downloads from Hugging Face or other hosted URLs.
• Google AI Core / system local AI integration. We are tracking Android's newer on-device AI stack so PokeClaw can eventually use official system-level local model APIs where they make sense, instead of relying on a single runtime path forever.
• More built-in workflows. More quick-task / skill coverage is planned beyond the first WhatsApp-centric workflows.
• Remote control / remote conversation flows. Controlling a phone from another device is a real request. Telegram bot control is the first concrete path being tested; it needs clear token setup, polling status, and true end-to-end QA with an account that can start and message the bot.
• Voice input. A microphone button is a useful prompt input path. The first version should be explicit push-to-talk; wake-word/background listening needs a separate privacy, permission, and battery design.

Known platform constraints

• Edge Gallery model detection is not fully under our control. Android hides other apps' Android/data/... sandboxes from normal file-pickers, so PokeClaw cannot generically "see" Edge Gallery's downloaded models unless they are exported into a user-accessible location first.
• GPU support is runtime and device specific. If a device can run GPU in another app but PokeClaw falls back to CPU, the next step is a debug report with device, ROM, model, selected backend, delegate init, and fallback logs. PokeClaw should keep CPU fallback safe and truthful instead of forcing GPU blindly.
• Sideload + accessibility apps may trigger OEM security warnings. Samsung / Play Protect warnings are being addressed through a cleaner release/signing path, but sideload trust prompts are partly controlled by the platform and OEM policy.
• Cloud mobile-agent harnesses already exist. PokeClaw should not compete as a cloud phone farm or desktop-controlled ADB framework. The durable product direction is a phone-resident Android harness that can run on the user's own device, slot local or cloud models, and survive real Accessibility, Notification, storage, permission, and OEM lifecycle constraints.
• Telegram bot control requires a writable account state. A Telegram bot cannot cold-message a user who has not started the bot, and a frozen/read-only Telegram account cannot complete the required /start or task-message step. Treat this as an environment blocker, not a PokeClaw channel failure.

External Automation API

PokeClaw exposes a user-enabled Android intent entrypoint for Tasker, MacroDroid, Locale, ADB, and other trusted local automation tools.

The setting is off by default. Enable it from Settings -> Remote Control -> External Automation.

MacroDroid verified E2E

The external automation contract is designed for Tasker, MacroDroid, Locale, and similar Android automation apps. The verified end-to-end path today is MacroDroid.

Verified on a Pixel 8 Pro on 2026-04-30:

• MacroDroid macro trigger: Shortcut Launched
• MacroDroid action: Send Intent
• Intent target: Activity
• Intent action: io.agents.pokeclaw.RUN_TASK
• Package: io.agents.pokeclaw
• Class: io.agents.pokeclaw.automation.ExternalAutomationActivity
• Extra: task = how much battery left
• Result: MacroDroid launched PokeClaw's automation activity, PokeClaw accepted it, executed the deterministic battery tool, and returned a live battery result in the PokeClaw chatroom.

MacroDroid setup:

1. In PokeClaw, enable Settings -> Remote Control -> External Automation. 2. In MacroDroid, create a new macro. 3. Add any trigger you want. For a simple smoke test, use User Input -> Shortcut Launched. 4. Add action Device Actions -> Send Intent. 5. Set Target to Activity. 6. Set Action to io.agents.pokeclaw.RUN_TASK. 7. Set Package to io.agents.pokeclaw. 8. Set Class to io.agents.pokeclaw.automation.ExternalAutomationActivity. 9. Set Extra 1 name to task. 10. Set Extra 1 value to the task text, for example how much battery left. 11. Save the macro, then run MacroDroid's Test macro.

Expected result: PokeClaw opens, records the task in the chatroom, runs the task through the normal harness, and shows the result. The intent does not bypass P