How to use
- Copy the installation command and review the source package before installing it in your own agent environment.
- Check the source document for required tools, credentials and supported hosts.
- Describe your input and desired deliverable; review the output against the source material.
npx skills add https://github.com/vercel-labs/agent-desktop --skill agent-desktop-windowsBefore you start
- Installation does not include a model subscription, API allowance or permission to access your files.
- Repository stars and installation counts describe activity, not an independent quality or security test.
Rendered from the source repository. Headings, examples, code, tables, links, and referenced images are preserved.
agent-desktop-windows
The observe-act loop, ref system, JSON envelope, sessions and tracing are the same on every platform. This file covers only what differs on Windows.
Requires Windows 10 1809+ / Windows Server 2019+ (x64 or ARM64). Run it in the interactive desktop session of the signed-in user. Session 0 (services and SYSTEM tasks), Server Core, the secure desktop (UAC prompts, the sign-in screen), a locked desktop and another user's session are not supported: observation, actions and capture there fail or return nothing.
Capability Table
| Group | Commands | Status on Windows |
|---|---|---|
| Observation | snapshot, find, get, is, screenshot | Works. Surfaces are window, focused, a Chromium modal as sheet, and an open application menu as menu |
| Surfaces | list-surfaces | Works. Lists window, focused and sheet per process, plus menu with item_count while a menu is open |
| Shell surfaces | open-system-surface | Works for start-menu, taskbar, system-tray, system-tray-overflow and action-center, and returns the window snapshot --surface <kind> reads. quick-settings returns PLATFORM_NOT_SUPPORTED naming action-center |
| Ref interaction | click, right-click, type, set-value, clear, select, toggle, check, uncheck, expand, collapse, scroll, scroll-to | Works, semantic in headless and --headed mode. right-click and type need --headed and return POLICY_DENIED headless |
| Multi-click and focus | double-click, triple-click, focus | Works with global --headed only |
| Keyboard and mouse | press, hover, drag, mouse-click, mouse-move, mouse-wheel | Works. Cursor-moving input needs --headed. First Contact covers press --app |
| Held input | key-down, key-up, mouse-down, mouse-up | Fails closed on every platform with ACTION_NOT_SUPPORTED. A headless mouse-down or mouse-up returns POLICY_DENIED first |
| App and window | launch, close-app, list-windows, list-apps, focus-window, resize-window, move-window, minimize, maximize, restore | Works. First Contact covers the name launch takes |
| Displays | list-displays | Works |
| Clipboard | clipboard-get, clipboard-set, clipboard-clear | Works; typed text and image content |
| Wait | wait | Works for ms, --element, --window, --text, --menu, --menu-closed, --event and --notification |
| Notifications | list-notifications, dismiss-notification, dismiss-all-notifications, notification-action | Works over the Action Center. Pass --headed when the center is closed |
| Cursor overlay | cursor-overlay | Works. enable returns data.rendered: true only after the renderer acknowledges over its pipe. See references/shell-and-overlay.md |
| System | status, permissions, version, batch, skills, session, trace | Works |
First Contact
- Quote every ref in PowerShell. PowerShell reads a bare
@tokenas its
splatting operator and deletes the argument before the binary sees it, so set-value @s8f3k2p9:e1 hi fails INVALID_ARGS. Write set-value '@s8f3k2p9:e1' hi. cmd.exe and bash need no quoting.
- A window behind another window is drivable. A minimized one is not.
Every element of a minimized window reports offscreen and ref actions fail. Run restore --app <image> and snapshot again. Do not use focus-window by reflex: it steals the user's foreground.
- No permission dialog exists.
permissionsreportsautomationas
not_required. Elevation boundaries are the only access control.
- Chromium and Electron apps read thin before they settle. Pass a longer
--timeout-ms to snapshot instead of concluding the tree is empty.
- Blocked shortcuts.
alt+f4,win+l,win+d,alt+tab,win+r,
win+x, win+e, ctrl+shift+esc and their modifier supersets are refused without --force. In combos, cmd and meta are the Windows key, so meta+c is a shell shortcut. Write ctrl+c to copy.
- `type` needs `--headed`. It is keystroke synthesis through
SendInput,
so a strict-headless type returns POLICY_DENIED. Use set-value, or pass --headed. It ends delivered_unverified (verification_scope: "unavailable") when the field exposes no readable Text selection, and can verify when one is readable: confirm with a fresh get.
- `press --app` writes to the foreground queue with no per-process
targeting. A headless press to a non-frontmost app fails closed. A non-interactive caller (service, scheduled task, CI job) gets delivered_unverified.
- `launch` takes an absolute path or a bare name under System32 or the
Windows directory, with or without .exe (notepad or notepad.exe), not a display name. Use list-apps to find a running app.
- Notifications:
list-notificationswith the Action Center closed needs
--headed. Single-item dismiss-notification and notification-action need --headed and --expected-app or --expected-title for the entry at INDEX.
- Surfaces: an open or context menu is the
menusurface. Windows does
not advertise a menubar surface.
- Commands that raise shell chrome take the foreground. Strict headless
refuses them with POLICY_DENIED before anything is raised.
- Stateful actions run once, then verify by re-reading.
delivered_verified
means the readback matched. ACTION_FAILED with details.kind: "post_action_verification" means the action landed but the readback differs: read details.post_state and do not retry. A set-value the app reformats reads as failed. See references/troubleshooting.md.
- Unsigned binary warnings are expected. See
references/troubleshooting.md.
What differs on Windows
- Ref actions are semantic UI Automation calls in headless and
--headedmode, exceptright-click,double-click,triple-click,typeandfocus, which are physical or take focus and need--headed. - The cursor overlay plays its per-action travel only for headless semantic actions. A
--headedaction moves the real cursor instead.
References
references/shell-and-overlay.md: open it foropen-system-surface, tray
and taskbar surfaces, notifications, the cursor overlay, hosted (UWP) window identity and menu detection coverage.
references/permissions-and-elevation.md: integrity levels, UIPI, blocked
combos, protected processes, the cross-process interaction lease.
references/chromium-and-electron.md: settle timing, covered-window hazard,
WPF wrong-provider trap, unnamed web content.
references/troubleshooting.md: symptom-to-cause map, action verification
outcomes, saving a document headless, install warnings.
Read any file from the binary: agent-desktop skills get agent-desktop-windows references/troubleshooting.md.
