Cua adds pixel-based perception for safer desktop automation

Cua has introduced an optional perception layer designed to improve safety during desktop automation. The system processes screenshots and converts them into labeled action regions. Instead of relying on fixed screen coordinates, it binds click actions to fresh, time-limited captures of the interface. This approach reduces the risk of an agent interacting with an element that has moved or changed between the moment of analysis and the moment of execution.

The method is particularly relevant for software agents that operate on canvases or applications without accessibility trees. In these environments, traditional element selectors are often unavailable, making pixel-based perception a practical alternative. By regenerating captures for each action, the layer creates a short-lived binding that limits the window for misdirected input.

This design offers a safety pattern for teams deploying automation across fields such as IT and development, product development, operations, and other technical domains where reliable GUI interaction is required. The feature is available as part of the Cua project.

Source: https://github.com/