Year: 2025
Type: conference-paper
Abstract: This paper introduces Hidden Forms (hFORMS), a dataset of natural language commands paired with user interfaces with masked visual context.By obscuring specific UI elements, the dataset challenges Computer-Using Agents to parse natural language instructions and infer the correct bounding box locations by leveraging UI context.Furthermore, hFORMS contains three distinct masking strategies representing progressive difficulty levels.Additionally, we explore parameter-efficient fine-tuning approaches using Vision-Language models from the Llama and Qwen series, demonstrating that fine-tuning on mobile domains results in more than 5x improvement in zero-shot domain adaptation performance when identifying bounding boxes on the desktop and web domains.
Authors: Anirudh Sundar, Christopher Gordon Richardson, William Gay, Benjamin Reichman, Larry Heck
Institutions: Georgia Institute of Technology, Google (United States)
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Award: G1452086972
OpenAlex ID: W4412944168