desktop_ui_automation v1.0.0

name: desktop_ui_automation description: "Automate native desktop apps that agents cannot reach by API: pick the right control method (accessibility tree first, image/OCR as fallback, keyboard last), build waits and retries that hold, and know when to stop and go manual. Desktop-native only; nothing here touches the browser." metadata: { "includeInPrompt": true }

Desktop UI Automation

Purpose

Some desktop apps have no API, no CLI, and no web interface. The only way in is the same way a person uses them: windows, buttons, menus. This skill covers the decision layer for that work: which control method to try first, how to make each attempt reliable instead of flaky, and when to stop and hand the job back to a person.

Scope: native desktop applications only, outside the browser. For in-browser automation (tabs, forms, page content), use the browser-task-patterns skill instead. This skill assumes you already checked for an API or CLI and found none.

Workflow

1. Choose the control method, in this order

Start at the top and step down only when the level above fails. Never start at coordinates.

Level 1: Accessibility tree. Query the OS accessibility layer directly. This is the most reliable method because you address real controls, not pixels. Use the tool that matches the platform:

• Windows: Windows UI Automation through pywinauto (pip name pywinauto) or the uiautomation package (pip name uiautomation). Both were verified present on PyPI (pywinauto 0.6.9, uiautomation 2.0.29) as of this writing.
• macOS: the Accessibility API through atomac (pip name atomac, verified on PyPI at 1.1.0) or AppleScript via the pyobjc bridge (both documented, neither executed here).
• Linux: AT-SPI through dogtail (pip name dogtail, verified on PyPI at 2.1.0) or the distro package python3-pyatspi (documented; note the bare pyatspi pip name returned no results on PyPI, do not invent that install path).

Prefer stable identity first: AutomationId or equivalent unique ID, then the accessible name, then control type plus position in the tree. Name matching is brittle when apps get translated or relabeled, so treat a bare name match as a hint, not a contract.

Level 2: Image matching plus OCR. Take screenshots and find controls visually. Use pyautogui.locateOnScreen for template matching against a saved reference image, and OCR with pytesseract (the Python wrapper, pip name pytesseract, verified on PyPI at 0.3.13) over the tesseract binary to read text off the screen. This level is honest but fragile: theme changes, resolution changes, and window repositioning all break it. Keep reference images current and always search relative to the app window, never the whole screen when you can avoid it.

Level 3: Keyboard navigation. When a control exists but nothing can see it, drive the keyboard: Tab through controls, Alt-key menu shortcuts, arrow keys, Enter and Escape. Use pynput (pip name pynput, verified on PyPI at 1.8.2) or pyautogui's keyboard functions. Keyboard flows are surprisingly durable across layout changes, and they work through most remote-desktop or VM walls where screenshots and the accessibility tree both die. Record the exact key sequence, since you will replay it blind.

Level 0 (the real first step): skip all of this. If the app has a config file, a registry key, a hidden command-line flag, or a documented automation hook, use that instead of the GUI. Ten minutes of research on the vendor docs beats a week of pixel automation.

2. Inspect before acting

At Level 1, enumerate the tree first: list the window's controls with their names, types, and IDs before you click anything. On Windows, pywinauto can dump the control tree to text; on macOS and Linux use the same listing the platform's accessibility inspector gives you. Save that dump with the job. You cannot debug what you never looked at.

At Levels 2 and 3, take a screenshot before the first action and keep it. State your target out loud in the plan: "the Save button in the File menu of the settings dialog", not "the thing at (412, 88)".

3. Make waits explicit

Fixed sleeps are where desktop automation goes to die. Replace them:

• Wait for a window or control to exist before touching it (pywinauto's wait family exists for this reason).
• After an action, wait for the expected state change: a new window title, a button becoming enabled, a dialog closing. State checks, not timers.
• Cap every wait with a timeout and a clear failure message that names what never appeared.

A minimal retry helper for Level 2 work is inlined below. Save it as desktop_click.py and run it with python3 desktop_click.py after filling in the target image path and the state check. Status: verified. import pyautogui (0.9.54) and pyautogui.size() executed locally under Xvfb; the wait/retry/click logic above was extracted verbatim from this file and executed, with only screen capture stubbed (unavailable headless): first poll missed, second matched, one click fired at the located center. Platform-specific Level 1 libraries below are documented from PyPI/source records, not executed on their native platforms.

python
# desktop_click.py - save this file, then run: python3 desktop_click.py
import time
import pyautogui  # pip install pyautogui (verified present on PyPI, 0.9.54)

TARGET_IMAGE = "save_button.png"   # your reference screenshot of the control
TIMEOUT_SEC = 30
TRIES = 3

def wait_and_click(image, timeout=TIMEOUT_SEC):
    """Click the first match for `image`, retrying until timeout. Returns True on click."""
    deadline = time.time() + timeout
    while time.time() < deadline:
        match = pyautogui.locateOnScreen(image, confidence=0.9)
        if match:
            pyautogui.click(pyautogui.center(match))
            return True
        time.sleep(0.5)
    return False

def main():
    for attempt in range(1, TRIES + 1):
        if wait_and_click(TARGET_IMAGE):
            print(f"clicked on attempt {attempt}")
            return
        print(f"attempt {attempt}: no match, retrying")
    raise SystemExit(f"gave up after {TRIES} attempts: {TARGET_IMAGE} never appeared")

if __name__ == "__main__":
    main()

4. Verify after every action

Each click is a claim, and claims get checked. After acting, confirm the world changed the way you expected: read the control's new value, check the window title, look for the success dialog. If the state check fails, do not barrel forward; back up or stop. An unverified action mid-sequence turns one failure into three.

Screenshot on every failure. A run that failed with no screenshot is a run you will have to redo to debug.

5. Retry with judgment, then stop

Retry the step, not the whole job. When a step fails:

1. Retry the same action once, after re-waiting for its precondition.
2. If it fails again, step back one level in the control hierarchy (Level 1 -> Level 2 -> Level 3) and retry once with the dumber method.
3. If that fails, stop the run. Do not add more sleeps and loop forever.

Give-up criteria. Hand the job back to a person when any of these is true:

• The app exposes no accessible names or IDs, the layout shifts between runs, and the visual match confidence keeps dropping. Three levels all tried and failed is a verdict, not a streak to break.
• You cannot reliably get the target window to the foreground, or clicks land on whatever window happens to be on top. Automation without focus control is chaos with a script.
• The flow passes through a human-verification gate (CAPTCHA-style check, biometric prompt, hardware token). Do not route around these.
• A step is destructive (delete, overwrite, send, pay) and you cannot verify the target before acting. Destructive plus unverifiable means manual.
• The same failure repeats across three separate sessions with different causes each time. That is the app telling you the surface is unstable.

When you give up, deliver a handoff note: what the goal was, which level each step reached, the last good state with a screenshot, and the exact manual steps remaining. A clean handoff is a success, not a failure.

Output Contract

A desktop automation run should produce, at minimum:

• The method chosen and why (which level, which library).
• The control dump or reference images used to target controls.
• A step log: action taken, state check performed, result.
• Screenshots of any failure.
• Either a completed task with its final state confirmed, or a handoff note with the last good state and remaining manual steps.

Operating Rules

1. Accessibility tree first, image matching second, coordinates never as a plan. A coordinate that appears in a plan must carry the window anchor and DPI it was measured at, or it does not ship.
2. Never click or type without confirming the target window is active and on top. Check focus, do not assume it.
3. Waits name their condition and their timeout. A bare sleep(5) in a script is a bug report waiting to be filed.
4. Verify after every action. An unverified click is a guess, and guesses do not compose.
5. Destructive actions get a screenshot taken immediately before, and a state check immediately after, every time. No exceptions.
6. Keep the app in a known starting state: launch it fresh or restore the window before the run, and close what you opened when done.
7. Remote-desktop and VM walls change the rules: accessibility trees and pixel matching usually die through them. Keyboard navigation and config-file automation survive. Plan for the wall before you build for the desktop.
8. When a commercial RPA suite (UiPath, Automation Anywhere, and similar) comes up as an alternative, name it as an option only and label pricing and capability claims "confirm before quoting". This skill does not endorse or price any commercial tool.
9. Stay on the desktop. If the target turns out to be a web page inside a browser window, switch to the browser-task-patterns skill; the layers in this skill are the wrong tools for that surface.