software-mansion/argent

argent-device-interact

Interact with an iOS simulator or iPhone, Android emulator or phone, or Chromium (CDP) app using argent MCP tools.

View source

How to use

  1. Copy the installation command and review the source package before installing it in your own agent environment.
  2. Check the source document for required tools, credentials and supported hosts.
  3. Describe your input and desired deliverable; review the output against the source material.
npx skills add https://github.com/software-mansion/argent --skill argent-device-interact

Before you start

  • Installation does not include a model subscription, API allowance or permission to access your files.
  • Repository stars and installation counts describe activity, not an independent quality or security test.
Original skill document

Rendered from the source repository. Headings, examples, code, tables, links, and referenced images are preserved.

Each tool's own description carries its parameters, defaults, return shape and examples. This skill covers choosing between tools and the gotchas those descriptions don't state.

1. Before You Start

  • Get a target id from list-devices; if none is ready, boot-device one. Full setup flow: argent-ios-simulator-setup / argent-ios-device-setup / argent-android-emulator-setup. The same tool names drive iOS, Android and Chromium; each takes the udid from list-devices.
  • Physical iPhone (kind: "device"): read argent-ios-device-interact first. Only the tools it lists exist there, and describe sees only the app launch-app registered.
  • Mac app (platform: "macos"): read argent-macos-app-interact first.
  • Read the matching reference before driving these targets: references/chromium.md for a chromium target, references/harmonyos.md for a harmony target, references/foldables.md for a device list-devices marks foldable: true (or before calling fold).
  • Load a deferred tool's schema with ToolSearch before its first call (e.g. gesture-pinch, gesture-rotate, gesture-custom); without it, numbers go out as strings and fail validation.
  • If you delegate device tasks to sub-agents, make sure they have MCP permissions.

2. Choosing the Right Tool

ActionToolNotes
Several known stepsrun-sequenceOne call, one screenshot at the end (§5)
Open an applaunch-appNever tap home-screen icons
Restart an apprestart-appClean in-memory state
Open URL / deep linkopen-urlWeb pages, deep links, URL schemes
Tapgesture-tapButtons, links, checkboxes
Scroll / swipegesture-swipeTouch targets; not Chromium
Scroll (Chromium)gesture-scrollMouse wheel
Drag (Chromium)gesture-dragSliders, drag-and-drop, text selection
Long press, drag & dropgesture-customAlso any other non-linear touch sequence
Pinch / zoomgesture-pinch
Two-finger rotationgesture-rotate
Hardware keybuttonHome, back, power, volume, appSwitch, actionButton
Type text or press a named keykeyboardEvery platform; text or one key per call (§6)
Paste textpasteOnly where a user would paste (§6). Sim/emu, Mac app
Rotate devicerotateOrientation changes
Fold devicefoldFoldables only: references/foldables.md
Shake deviceshakeShake handlers (sim/emu only), Undo-typing prompt, RN dev menu
Wait for UIawait-ui-elementBlock until an element is visible/hidden/exists/contains text
Wait for idleawait-screen-idleBlock until a non-empty screen tree stops changing (§4)

3. Finding Tap Targets

After an action moves you to a different screen, or whenever you don't know an element's coordinates, discover before tapping. Interaction results already carry the element tree (--- Elements after action (describe) ---): use it, and call a discovery tool only when no fresh tree covers the current screen. Each tree has a Tree id. When the tree is identical to an earlier one, the result names that id in place of the tree: use that tree, and call describe if it is not in your context. An identical tree does not mean the action did nothing.

NeedTool
Any screen: app, system dialog, Home screendescribe
UIKit properties of a view (iOS simulator)native-find-views (needs bundleId and a class, id, label, tag or nativeID to match)
What sits at a known raw point (iOS simulator)native-view-at-point (the visually deepest view; hitTest: true for the view that receives touch)
Final visual fallbackscreenshot, only when the tools above cannot expose the control. Do not derive routine in-app navigation targets from pixels

If describe returns an empty tree, follow its hint; with no hint, the screen is blank, loading or unlabeled: check the attached screenshot and retry once content has loaded.

For a React Native testID or component name that describe does not show, use debugger-component-tree; take tap points from describe.

React Native: LogBox toasts sit over the bottom of the screen and catch taps meant for what is under them. Argent hides them when a debugger-* tool connects and after debugger-reload-metro; after restart-app, call a debugger-* tool again. After any other reload they come back: close each by tapping the right end of its describe frame (the X), not its middle.

System and permission dialogs: on a simulator or emulator, describe returns a visible dialog's buttons with tap coordinates; keep using it for in-app modals too. Fall back to screenshot only if it does not expose the controls: then tap one obvious button (Allow, OK, Don't Allow, Not Now, Continue) at a time, check the returned screenshot, and go back to describe once the dialog is gone.

Prefer the dialog over the Settings tool. When the app triggers its own permission prompt, answering it is the real user path. Use settings-permissions only when you can't make the change through the app: set a permission before the app asks, re-enable one the user denied (iOS won't re-prompt), or reset it so the prompt reappears. See argent-settings-permissions.

4. Waiting: await-ui-element and await-screen-idle

Never poll `screenshot`/`describe` in a loop to wait for something. await-ui-element blocks server-side on the same tree describe reads. It has no bare-timer mode; for a plain pause, use your own harness sleep. Its description covers conditions, selector matching and defaults. What it does not tell you:

  • A hidden check that succeeds immediately may be a false pass: its note then says the selector never matched anything. Treat that as a failed check and fix the selector; it does not mean "the element went away".
  • The synthetic ROOT container describe prints is never matched, so a role like AXGroup/html won't trivially "match the screen".
  • To disambiguate a loose selector, pin role to a text role like StaticText; that skips a same-named button.

await-screen-idle is for live diagnosis after a launch or navigation, before a raw tap on an element that may still be moving. Continue only when settled: true, and pair it with an await-ui-element for the destination: stillness does not identify a screen, and the tool can return during a presentation-layer animation. Do not record it in an e2e test; tests use await: { idle: true }, which also compares pixels.

5. Action Sequencing with run-sequence

Use run-sequence when every step is known in advance; if a step depends on what the previous one shows, use separate calls. Its description lists the allowed tools and their arguments. Typical batches:

  • "scroll to bottom", "scroll until X": 3-5 swipes
  • "clear and retype a field": triple-tap to select all, then type the new value
  • typing then submitting: a keyboard text step, then a keyboard enter step (one call cannot carry both)
  • "submit form": fill every field, then tap submit
  • "go back to X": a known tap path

Gate a step that depends on a screen transition with an await-ui-element step rather than a fixed delayMs. If its condition is not met before the timeout, the sequence stops there and the later steps do not run, so a tap can't fire against a screen that never appeared. Any failing step also stops the sequence, with partial results.

6. Text Entry and Secrets

  • Tap the field first. Typing or pasting with no focused field is a silent no-op.
  • paste is not a faster keyboard. keyboard types the way a user types and stays the default for every text entry. Use paste only where a real user would paste: a 2FA / OTP code from another app, a long link or token, or testing how the app handles pasted input. That it can carry text keyboard can't type on a platform (multi-line text, non-ASCII on Android) is not a reason on its own.
  • Credentials: type them as a {{secret:<NAME>}} placeholder in text (keyboard, paste, run-sequence keyboard steps, e2e type steps), never as plaintext, and never ask the user for the value. Do not describe or screenshot a non-secure field you just filled with a secret: submit or navigate away first, then verify the resulting screen.

7. Screenshots

Interaction tools already attach a screenshot. Call screenshot for the initial state, for a baseline before you change visible UI, to check state after a delay, or when the attached capture shows a transitional or loading frame.

Keep the default downscaled capture for UI context and state checks; do not use scale: 1.0 as a readability or tapping aid. Full resolution is only for saving baseline/current PNGs for comparison (scale: 1.0 with includeImageInContext: false); for visual regression checks use the argent-screenshot-diff skill.

If screenshot times out, restart the device transport with stop-simulator-server.

8. Android notes

  • First-launch permission prompts: reinstall-app on Android installs with -g, so runtime permissions are pre-granted on first launch.
from this repository

More skills

All skills
software-mansion
Community

argent-android-emulator-setup

Set up and connect to an Android emulator using argent MCP tools. Use when starting a new session on Android, booting an emulator, getting a device serial, or before any UI interaction task.

1h installs
6
GitHub stars
3.2K
Updated
Nov 19
software-mansion
Community

argent-create-e2e

Create, record, edit, replay, or repair reusable Argent e2e test YAML files. Use when the user asks to record, replay, or run an e2e test or a repeatable device path, set up profiling or an A/B comparison, or invoke the authoring engine behind argent-qa-e2e. Also use before repeating three or more interactions. For one-off UI checks, acceptance-driven regression tests, or screen video, use argent-test-ui-flow, argent-qa-e2e, or argent-screen-recording respectively.

1h installs
6
GitHub stars
3.2K
Updated
Nov 19
software-mansion
Community

argent-ios-device-interact

Drive a physical iPhone through argent. Use when a physical iPhone is involved (a list-devices iOS entry with kind "device") and ONLY then; never for a simulator. For cable, trust, or signing problems read argent-ios-device-setup.

1h installs
6
GitHub stars
3.2K
Updated
Nov 19
software-mansion
Community

argent-ios-device-setup

Set up a cabled physical iPhone for argent. Use when a physical iPhone is involved (a list-devices iOS entry with kind "device") and ONLY then; never for a simulator.

1h installs
6
GitHub stars
3.2K
Updated
Nov 19