Work

Associating the visual representation of user interfaces with their internal structures and metadata

Year: 2011

Type: conference-paper

Abstract: Pixel-based methods are emerging as a new and promising way to develop new interaction techniques on top of existing user interfaces. However, in order to maintain platform independence, other available low-level information about GUI widgets, such as accessibility metadata, was neglected intentionally. In this paper, we present a hybrid framework, PAX, which associates the visual representation of user interfaces (i.e. the pixels) and their internal hierarchical metadata (i.e. the content, role, and value). We identify challenges to building such a framework. We also develop and evaluate two new algorithms for detecting text at arbitrary places on the screen, and for segmenting a text image into individual word blobs. Finally, we validate our framework in implementations of three applications. We enhance an existing pixel-based system, Sikuli Script, and preserve the readability of its script code at the same time. Further, we create two novel applications, Screen Search and Screen Copy, to demonstrate how PAX can be applied to development of desktop-level interactive systems.

Source: ACM Symposium on User Interface Software and Technology (UIST)

DOI: 10.1145/2047196.2047228

Authors: Tsung-Hsiang Chang, Tom Yeh, Rob Miller

Institutions: Massachusetts Institute of Technology, University of Maryland, College Park

Language: English

Topic: Usability and User Interface Design

Subfield: Human-Computer Interaction

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

Funder: National Science Foundation

Award: CAREER: End-user Programming for Web Automation and Visualization