agent-device: Give AI Coding Agents a Mobile App Feedback Loop

The short answer

agent-device is an open-source CLI, MCP server, and typed Node.js API for letting AI coding agents inspect, operate, debug, and verify running applications. Its practical value is not merely automated tapping: it gives an agent a compact accessibility-tree snapshot, stable interaction references, device sessions, replayable workflows, and reviewable evidence such as screenshots, logs, traces, and performance samples.

That makes it a useful missing layer in mobile AI-assisted development. A coding agent can modify a React Native, Expo, native iOS, or Android app, open the running build, exercise a flow, and return evidence instead of stopping after a successful typecheck or unit test.

For most teams, the safest adoption path is:

1. Install a reviewed, stable agent-device version in the project or developer environment.
2. Start with a simulator or emulator and a narrow verification task.
3. Prefer accessibility snapshots and selectors for actions, with screenshots as evidence.
4. Keep device access scoped to the project and close sessions after each run.
5. Promote successful explorations into replayable CI checks only after a human reviews them.

What agent-device actually adds to an AI coding workflow

Traditional coding-agent loops are strongest when the feedback is textual: compiler output, tests, lint results, and repository diffs. Mobile applications add a difficult final mile. The agent must see a live screen, handle permissions and keyboards, interact with gestures, observe network or native failures, and confirm that the result is usable on a device.

The official agent-device repository describes one runtime exposed through three entry points:

• A terminal CLI for agents that already work through shell commands.
• An official stdio MCP server for clients that prefer structured tools.
• A typed Node.js client for embedding device operations in an agent or QA product.

The project supports iOS, Android, and HarmonyOS, as well as TV, web, macOS, and Linux targets with platform-dependent capability depth. It can work with simulators, emulators, physical devices, and selected remote device clouds.

The important design choice is semantic inspection. Instead of requiring an agent to reason from screenshots alone, snapshot -i exposes an accessibility-oriented representation with roles, labels, values, and references such as @e2. The agent can act on a current reference, settle the interaction, inspect the resulting diff, and then verify the state.

The agent-device feedback loop

Visual: original workflow diagram based on the session model and inspect–act–verify guidance in the official agent-device README and AI Agent Setup documentation.

This is more useful than “AI-generated end-to-end tests” as a slogan. The agent has a defined observation and evidence path, while the developer retains control over which environments, devices, credentials, and actions are allowed.

Installation and a first verification run

The official quick start uses a global installation:

1npm install -g agent-device@latest
2agent-device doctor
3agent-device help workflow

For unattended or team workflows, do not blindly let an agent fetch a mutable package at runtime. The project documentation explicitly recommends a trusted installed binary or a project-local, lockfile-pinned version for normal agent use. A human should review the chosen version and the permissions needed by the target platform.

For an Expo project, Expo’s agent-device guide says that no agent-device library needs to be added to the application itself. The tool operates the installed running app. A basic development loop can look like this:

1npx expo run:android
2
3agent-device open MyApp --platform android
4agent-device snapshot -i
5agent-device press @e2 --settle
6agent-device screenshot ./artifacts/after-press.png
7agent-device close

Reference identifiers are intentionally short-lived. After a settled interaction, use the refs in the returned diff; do not assume that @e2 still identifies the same element after the UI changes. For durable replay, prefer selectors, accessible labels, or test IDs that your application exposes consistently.

The same inspect–act–verify pattern applies to a form flow:

1agent-device open Checkout --platform ios
2agent-device snapshot -i
3agent-device fill @e7 "Ada" --settle
4agent-device fill @e9 "Lovelace" --settle
5agent-device press @e12 --settle
6agent-device assert "text=Order confirmed"
7agent-device screenshot ./artifacts/checkout-confirmed.png
8agent-device close

The exact refs and supported commands vary by platform and app state. Treat these snippets as workflow shapes, not universal coordinates.

CLI, MCP, or Node.js API?

The official MCP configuration is intentionally small:

1{
2  "mcpServers": {
3    "agent-device": {
4      "command": "agent-device",
5      "args": ["mcp"]
6    }
7  }
8}

Use MCP when the client can launch the reviewed local binary and the team wants structured discovery. Use the CLI when terminal-visible commands, logs, and explicit approvals are more important. Do not use the MCP server as a backdoor for arbitrary command execution, and do not silently fall back to npx -y agent-device@latest when the binary is missing.

For a custom agent, the repository documents a typed Node.js surface. The conceptual shape is:

1import { createAgentDeviceClient } from "agent-device";
2
3const client = createAgentDeviceClient({ session: "qa-run" });
4
5try {
6  await client.apps.open({
7    app: "com.example.app",
8    platform: "android",
9  });
10
11  const snapshot = await client.capture.snapshot({ interactiveOnly: true });
12  const button = snapshot.nodes.find((node) => node.role === "button");
13
14  if (button) {
15    await client.interactions.press({ ref: button.ref });
16  }
17} finally {
18  await client.sessions.close();
19}

Verify method names and platform options against the installed package’s version-matched API documentation before copying this into production. The architectural point is the same: open one session, inspect current state, act on a semantic target, verify, and close.

Why accessibility snapshots beat screenshot-only automation

Screenshots are valuable evidence, but they are a fragile primary control surface. A screenshot forces the agent to infer text, roles, hit regions, and state from pixels. An accessibility snapshot can expose a button’s role and label directly, reducing ambiguity and making failures easier to diagnose.

This depends on application quality. Missing accessibility labels, ambiguous roles, unstable test IDs, and custom-drawn controls make any agent workflow less reliable. Improving accessibility therefore improves both human usability and agent verification. Keep screenshots and video for visual regressions, layout checks, and review artifacts—not as the only way to decide what to tap.

The Expo documentation also describes compact semantic responses, actionable refs, and artifact handling as ways to preserve the agent’s context window. That is an important engineering trade-off: the verification tool should return enough state to make the next decision without dumping an entire device log into the prompt.

Debugging, profiling, and replay

A useful verification tool must explain failure, not only report that a tap failed. According to the official documentation, agent-device can route workflows toward focused logs, network activity, screenshots, video, traces, performance samples, crash details, and React Native inspection. Support varies by target, so check the installed version’s capability and debugging guides.

A practical escalation ladder is:

1. Reproduce the flow with an interactive snapshot.
2. Capture the smallest screenshot or video that proves the visible failure.
3. Open a focused log window rather than collecting an unbounded device log.
4. Add network, trace, or React profiling evidence only if it can distinguish the cause.
5. Save the successful or diagnostic path as a replayable .ad script when repetition matters.

The Callstack conference talk on mobile feedback loops provides independent practitioner context: mobile agents need a feedback loop over the running app, not just source-code access. It highlights accessibility-tree interaction, evidence collection, debugging, and turning exploratory work into repeatable checks.

Replay should be treated as reviewed test material. An agent may discover a path that works once but contains accidental state, weak assertions, or unsafe data. Human-review the generated flow, remove secrets and personal data, make the assertions explicit, and run it against a clean environment before putting it in CI.

Security and operational controls

Giving an agent control of a device is materially more sensitive than giving it read-only repository access. Use defense in depth:

• Use a dedicated simulator, emulator, or test device with synthetic accounts.
• Keep production credentials, payment cards, personal messages, and unrelated apps off the target.
• Scope MCP and CLI access to the project and approved device pool.
• Pin the package version and review updates before rollout.
• Keep mutating actions serial within one session.
• Require explicit approval for destructive actions, installs, account changes, or external communication.
• Store screenshots, videos, logs, and traces with appropriate retention and access controls.
• Close sessions, and shut down ephemeral emulators in CI when the job finishes.
• Make assertions narrow enough that a visually similar but incorrect screen cannot pass.

The project’s security policy and documentation’s setup guidance should be part of the review checklist. The tool can make a coding agent more autonomous, but it does not make the agent trustworthy by default.

Common failure modes

The agent taps the wrong element

Improve accessible names, roles, and test IDs. Re-snapshot after every state-changing action and use current refs. Avoid coordinates unless the platform-specific workflow explicitly requires them.

The MCP server starts but the client cannot find the binary

The agent process may have a different PATH from the developer’s shell. Resolve the reviewed absolute binary path or configure the client with a project-local executable. Do not automatically download an unreviewed package.

A flow works locally but fails in CI

Compare the app build, OS version, permissions, emulator state, fonts, network fixtures, and device capabilities. Start with a clean, deterministic environment and retain the first failing artifact.

Screenshots pass while behavior is broken

Pair visual evidence with semantic assertions, network checks, or a state assertion. A screen can look correct while a request failed, a button is inaccessible, or navigation did not complete.

The agent over-collects diagnostics

Ask for focused evidence. Unbounded logs and recordings increase cost, slow the loop, and can expose unrelated sensitive data.

When to adopt it

agent-device is a strong fit when a team already uses AI coding agents and ships mobile or cross-platform interfaces where source-level tests do not prove the user-visible result. It is especially useful for React Native and Expo teams that want one workflow across local devices, simulators, CI, and remote device providers.

It is less compelling if the project has no runnable device environment, cannot isolate test data, or only needs static code generation. Start with one high-value flow—onboarding, checkout with fake payments, or a regression that is hard to reproduce—and measure whether the evidence shortens review time without increasing operational risk.

For broader agent design, pair this device feedback loop with the site’s harness engineering guide, its GitHub Copilot SDK guide, its MCP Apps guide, and its agent security controls coverage. The connection is straightforward: the harness decides what the agent may do, the runtime performs the work, and agent-device supplies evidence from the running application.

FAQ

Is agent-device an MCP server or a testing framework?

Both descriptions are incomplete on their own. It is a CLI-first device automation and verification runtime that also exposes an official MCP server and typed Node.js API. It can support exploratory work, debugging, profiling, and repeatable checks.

Does it require adding a library to an Expo app?

No. Expo’s guide says the tool operates the installed running app; it does not require an agent-device library in the application. Platform toolchains and permissions are still required for the chosen device environment.

Can it replace human mobile QA?

No. It can automate observation and collect evidence, but humans still define the risk boundaries, review new workflows, validate test data, and decide whether evidence is sufficient for release.

Should an agent use screenshots or accessibility snapshots?

Use accessibility snapshots, refs, and selectors for interaction whenever available. Use screenshots, video, and traces as contextual evidence and for visual or diagnostic checks.

Conclusion

The most useful contribution of agent-device is a concrete answer to a common AI coding-agent weakness: an agent can write mobile code faster than a human can manually inspect every resulting build. By combining semantic snapshots, controlled actions, session state, diagnostics, and evidence, it moves mobile development toward a closed feedback loop.

Adopt it conservatively. Pin the tool, isolate the device, start with one workflow, assert the behavior, and review every replay before CI. Used that way, agent-device is not a claim that agents can ship mobile apps unsupervised. It is a practical verification layer that lets them observe the software they are changing.

Sources and visual credits

• Callstack: agent-device repository and README — primary source for capabilities, CLI, MCP, Node.js API, supported targets, and examples.
• Callstack: AI Agent Setup — primary documentation for installation, client configuration, safety guidance, and workflows.
• Expo: agent-device and Expo — independent ecosystem documentation and Expo workflow confirmation.
• GitNation: Giving AI Agents Hands: Mobile Feedback Loops with Agent Device — independent practitioner and conference context.
• Original Mermaid workflow diagram in this article, based on the cited official session model.

Related reading

Continue exploring related AI engineering and developer tooling topics:

• GitHub Copilot’s September 2026 Model Retirements: A Safe Migration and Regression-Test Guide
• Spring Tools 5.2.0: Claude Code, Embedded MCP, and Safer Spring Boot Workflows
• Google Cloud Agent Registry: A Practical MCP Governance Guide