How to use
- Copy the installation command and review the source package before installing it in your own agent environment.
- Check the source document for required tools, credentials and supported hosts.
- Describe your input and desired deliverable; review the output against the source material.
npx skills add https://github.com/software-mansion/argent --skill argent-device-interactBefore you start
- Installation does not include a model subscription, API allowance or permission to access your files.
- Repository stars and installation counts describe activity, not an independent quality or security test.
Rendered from the source repository. Headings, examples, code, tables, links, and referenced images are preserved.
Each tool's own description carries its parameters, defaults, return shape and examples. This skill covers choosing between tools and the gotchas those descriptions don't state.
1. Before You Start
- Get a target id from
list-devices; if none is ready,boot-deviceone. Full setup flow:argent-ios-simulator-setup/argent-ios-device-setup/argent-android-emulator-setup. The same tool names drive iOS, Android and Chromium; each takes theudidfromlist-devices. - Physical iPhone (
kind: "device"): readargent-ios-device-interactfirst. Only the tools it lists exist there, anddescribesees only the applaunch-appregistered. - Mac app (
platform: "macos"): readargent-macos-app-interactfirst. - Read the matching reference before driving these targets:
references/chromium.mdfor achromiumtarget,references/harmonyos.mdfor aharmonytarget,references/foldables.mdfor a devicelist-devicesmarksfoldable: true(or before callingfold). - Load a deferred tool's schema with ToolSearch before its first call (e.g.
gesture-pinch,gesture-rotate,gesture-custom); without it, numbers go out as strings and fail validation. - If you delegate device tasks to sub-agents, make sure they have MCP permissions.
2. Choosing the Right Tool
| Action | Tool | Notes |
|---|---|---|
| Several known steps | run-sequence | One call, one screenshot at the end (§5) |
| Open an app | launch-app | Never tap home-screen icons |
| Restart an app | restart-app | Clean in-memory state |
| Open URL / deep link | open-url | Web pages, deep links, URL schemes |
| Tap | gesture-tap | Buttons, links, checkboxes |
| Scroll / swipe | gesture-swipe | Touch targets; not Chromium |
| Scroll (Chromium) | gesture-scroll | Mouse wheel |
| Drag (Chromium) | gesture-drag | Sliders, drag-and-drop, text selection |
| Long press, drag & drop | gesture-custom | Also any other non-linear touch sequence |
| Pinch / zoom | gesture-pinch | |
| Two-finger rotation | gesture-rotate | |
| Hardware key | button | Home, back, power, volume, appSwitch, actionButton |
| Type text or press a named key | keyboard | Every platform; text or one key per call (§6) |
| Paste text | paste | Only where a user would paste (§6). Sim/emu, Mac app |
| Rotate device | rotate | Orientation changes |
| Fold device | fold | Foldables only: references/foldables.md |
| Shake device | shake | Shake handlers (sim/emu only), Undo-typing prompt, RN dev menu |
| Wait for UI | await-ui-element | Block until an element is visible/hidden/exists/contains text |
| Wait for idle | await-screen-idle | Block until a non-empty screen tree stops changing (§4) |
3. Finding Tap Targets
After an action moves you to a different screen, or whenever you don't know an element's coordinates, discover before tapping. Interaction results already carry the element tree (--- Elements after action (describe) ---): use it, and call a discovery tool only when no fresh tree covers the current screen. Each tree has a Tree id. When the tree is identical to an earlier one, the result names that id in place of the tree: use that tree, and call describe if it is not in your context. An identical tree does not mean the action did nothing.
| Need | Tool |
|---|---|
| Any screen: app, system dialog, Home screen | describe |
| UIKit properties of a view (iOS simulator) | native-find-views (needs bundleId and a class, id, label, tag or nativeID to match) |
| What sits at a known raw point (iOS simulator) | native-view-at-point (the visually deepest view; hitTest: true for the view that receives touch) |
| Final visual fallback | screenshot, only when the tools above cannot expose the control. Do not derive routine in-app navigation targets from pixels |
If describe returns an empty tree, follow its hint; with no hint, the screen is blank, loading or unlabeled: check the attached screenshot and retry once content has loaded.
For a React Native testID or component name that describe does not show, use debugger-component-tree; take tap points from describe.
React Native: LogBox toasts sit over the bottom of the screen and catch taps meant for what is under them. Argent hides them when a debugger-* tool connects and after debugger-reload-metro; after restart-app, call a debugger-* tool again. After any other reload they come back: close each by tapping the right end of its describe frame (the X), not its middle.
System and permission dialogs: on a simulator or emulator, describe returns a visible dialog's buttons with tap coordinates; keep using it for in-app modals too. Fall back to screenshot only if it does not expose the controls: then tap one obvious button (Allow, OK, Don't Allow, Not Now, Continue) at a time, check the returned screenshot, and go back to describe once the dialog is gone.
Prefer the dialog over the Settings tool. When the app triggers its own permission prompt, answering it is the real user path. Usesettings-permissionsonly when you can't make the change through the app: set a permission before the app asks, re-enable one the user denied (iOS won't re-prompt), or reset it so the prompt reappears. Seeargent-settings-permissions.
4. Waiting: await-ui-element and await-screen-idle
Never poll `screenshot`/`describe` in a loop to wait for something. await-ui-element blocks server-side on the same tree describe reads. It has no bare-timer mode; for a plain pause, use your own harness sleep. Its description covers conditions, selector matching and defaults. What it does not tell you:
- A
hiddencheck that succeeds immediately may be a false pass: itsnotethen says the selector never matched anything. Treat that as a failed check and fix the selector; it does not mean "the element went away". - The synthetic
ROOTcontainerdescribeprints is never matched, so arolelikeAXGroup/htmlwon't trivially "match the screen". - To disambiguate a loose selector, pin
roleto a text role likeStaticText; that skips a same-named button.
await-screen-idle is for live diagnosis after a launch or navigation, before a raw tap on an element that may still be moving. Continue only when settled: true, and pair it with an await-ui-element for the destination: stillness does not identify a screen, and the tool can return during a presentation-layer animation. Do not record it in an e2e test; tests use await: { idle: true }, which also compares pixels.
5. Action Sequencing with run-sequence
Use run-sequence when every step is known in advance; if a step depends on what the previous one shows, use separate calls. Its description lists the allowed tools and their arguments. Typical batches:
- "scroll to bottom", "scroll until X": 3-5 swipes
- "clear and retype a field": triple-tap to select all, then type the new value
- typing then submitting: a
keyboardtext step, then akeyboardenterstep (one call cannot carry both) - "submit form": fill every field, then tap submit
- "go back to X": a known tap path
Gate a step that depends on a screen transition with an await-ui-element step rather than a fixed delayMs. If its condition is not met before the timeout, the sequence stops there and the later steps do not run, so a tap can't fire against a screen that never appeared. Any failing step also stops the sequence, with partial results.
6. Text Entry and Secrets
- Tap the field first. Typing or pasting with no focused field is a silent no-op.
pasteis not a fasterkeyboard.keyboardtypes the way a user types and stays the default for every text entry. Usepasteonly where a real user would paste: a 2FA / OTP code from another app, a long link or token, or testing how the app handles pasted input. That it can carry textkeyboardcan't type on a platform (multi-line text, non-ASCII on Android) is not a reason on its own.- Credentials: type them as a
{{secret:<NAME>}}placeholder intext(keyboard,paste,run-sequencekeyboard steps, e2etypesteps), never as plaintext, and never ask the user for the value. Do notdescribeorscreenshota non-secure field you just filled with a secret: submit or navigate away first, then verify the resulting screen.
7. Screenshots
Interaction tools already attach a screenshot. Call screenshot for the initial state, for a baseline before you change visible UI, to check state after a delay, or when the attached capture shows a transitional or loading frame.
Keep the default downscaled capture for UI context and state checks; do not use scale: 1.0 as a readability or tapping aid. Full resolution is only for saving baseline/current PNGs for comparison (scale: 1.0 with includeImageInContext: false); for visual regression checks use the argent-screenshot-diff skill.
If screenshot times out, restart the device transport with stop-simulator-server.
8. Android notes
- First-launch permission prompts:
reinstall-appon Android installs with-g, so runtime permissions are pre-granted on first launch.