Year: 2020
Type: dissertation
Abstract: In recent years, businesses have become increasingly aware of their need to automate previously manual business processes. Adoption rates of robotic process automation (RPA) tools have increased a lot, and this has generated a great deal of value for companies. But, alas, especially some older legacy applications are still extremely problematic from the point of view of RPA platforms, and they cannot often be interfaced in a same fashion as some more modern applications. This is especially lamentable, as it is often the older applications, which tend to be used in the more manual business processes. This thesis aims to build a foundation for building and evaluating commercializable computer vision (CV) object detection models, that resolve the issues faced by RPA applications that arise from the lack of operating system (OS) level user interface (UI) visual element data. This kind of models could be used as a basis for building completely new RPA tools, or for augmenting existing solutions. The research questions discussed by this study are as follows: 1. Is using machine learning models for the detection of user interface elements viable? 2. What are the implications to existing RPA applications? 3. Can we build a model that is able to generalize from data from another domain? This thesis discusses the theoretical background of object detection and machine learning, and presents three potential families of machine learning model architectures that could be used for the aforementioned tasks. These models were then trained and compared to the CV capabilities of a commercial RPA platform, UiPath. The results show that the most simple object detection models cannot compete with the current commercial offering. There is, however, a clear path to further research, and it is clear that building a object detection system for UI elements is viable. In addition, this thesis presents the first publicly available dataset of (manually) annotated desktop UI screenshot images, and a tool for generating artificial user interface data.
Source: Aaltodoc (Aalto University)
Author: Waltteri Vuorimaa
Language: English
Subfield: Management Information Systems
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W3154899117