Äänellä ja tekstillä ehdollistettu, ihmisen ja tekoälyn vuorovaikutuksessa tapahtuva abstraktin äänen synteesi
Year: 2023
Type: dissertation
Abstract: Recent trends in computational creativity research have drawn attention to multi- modal models relating data from two or more modalities, such as text, image and audio. Even though multimodal models have been demonstrated as a successful approach to text-conditioned image generation, such models have not been as studied for generative tasks in the audio domain. This work attempts to fill the gap by study- ing audio and text conditioned abstract sound synthesis based on the multimodal AudioCLIP model. By creating sound abstractions from user input, the studied synthesis algorithm aims to allow such human-computer co-exploration with artificial intelligence (AI) adaptable to artistic work. As a computational creativity support tool, the studied algorithm is among the few tools offering AI-based ideation for professional composers. This work evaluates qualitatively the suitability of the suggested abstract sound synthesis algorithm for co-creative ideation. While different approaches to abstract sound synthesis were compared with experiments, the quality of the synthesized sounds and their usefulness to artistic work were evaluated in a user study with professional composers. As one of the main findings, the research reveals that the AudioCLIP model is not effective enough for the studied multimodal generative task. However, the synthesis-by-optimization approach adapted from an exemplary study is able to create sounds that are interesting to professional composers. Ensuring stronger resemblance between user-provided input and generated results and providing users more control for steering the system is among the topics worth further research. In addition, further research is motivated by professional composers’ need for automatic tools to replace the manual work of idea generation from inspirational examples, as observed in the user study.
Source: Aaltodoc (Aalto University)
Author: Hassinen, Heidi
Language: English
Subfield: Music
Field: Arts and Humanities
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7113350551