Building my project's Visit at: https://teapawt.net/t.co/k2zHdh8pD4
Joined August 2025
- Tweets1.7K
- Following480
- Followers324
- Likes2.5K
Partly. In the AI agent, the element-click tool refuses to guess: if more than one UI element matches a name, it returns the candidate list with IDs and the model has to pick one explicitly, and disabled elements are rejected.
In scripted steps it's heuristic rather than confidence-based: matching is case-insensitive substring, interactive UIA controls (buttons/links) win over OCR text, and remaining ties go to the match closest to the cursor, optionally narrowed to a window, region or control type.
Windows OCR doesn't give per-word confidence, so there's no threshold yet. Honestly, exact-match scoring hadn't crossed my mind, but it's a great idea and I'll probably add it.
Scripted steps don't store screen coordinates. Each run finds the target again by its visible text: UI Automation first, OCR as a fallback, and an optional LLM-based semantic search after that. Steps can be limited to a window or a screen region, so a moved window doesn't break them. The AI agent takes a fresh screenshot after every action, checks whether the click actually changed anything, falls back from text to the UIA element list to coordinates, and re-plans with loop guards.