Year: 2021
Type: dissertation
Abstract: Describing the world around us with text or speech is a common task for humans. Yet computer science research so far has mostly focused on generating captions and descriptions for two-dimensional images instead of three-dimensional worlds. When interacting with the real world, this limitation may seem fine, since the main way we perceive the world is in the form of two-dimensional projections in each of our eyes. Virtual worlds in computers are stored in a three-dimensional representation, making more direct approaches of generating descriptions viable. Research and video games have shown that algorithms are more than capable of generating convincing virtual worlds. Putting both of these capabilities together, in this thesis a novel approach of generating scenes and textual descriptions from them at the same time is presented. Therefore a scene relationship graph is derived from a hand-designed graph grammar. From this graph an evaluation function is designed and constructed that checks whether a scene layout, e.g.an arrangement of existing annotated 3D models, satisfies the constraints of the graph. Then, a random initial scene layout is optimized according to this evaluation function. The optimization employed is a Markov chain Monte Carlo method called Metropolis Hastings with parallel tempering. A textual description of this scene layout is produced by feeding the scene relationship graph into a natural language generation pipeline. The system is capable of producing good results with accurate descriptions, however, the output quality is random and has high variance. Especially the pathway generation system tends to produce poor layouts. A user study with 30 participants was conducted to evaluate scene quality and description accuracy. Each participant evaluated five scenes. In two thirds of the ratings scenes were deemed plausible and similarly in two thirds of the ratings scenes were also deemed aesthetically pleasing, with a strong correlation between the two groups. Participants also compared two descriptions each for twelve scenes. Some of these descriptions were user provided and some generated. The average rated accuracy of automatically generated descriptions was on average similar to the human descriptions. Automatically generated descriptions were even subjectively preferred by participants over human descriptions in half of the cases. However, the machine descriptions are not competitive with the human descriptions that were rated most accurate. These descriptions were shorter and more accurate in almost all cases. The experiments and user studies showed that a combined generation of scene layouts and textual descriptions is a viable strategy to produce reasonably good scene descriptions.
Author: Müller-Huschke, Julian
Institution: Universität Innsbruck
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7036220411