Avatar Fusion Karaoke: Research and development on multi-user music play VR experience in the metaverse
Year: 2023
Type: conference-paper
Abstract: This paper contributes to building a standard process of research and development (R&D) for new user experiences (UX) in metaverse services. We tested this R&D process on a new UX proof of concept (PoC) for Meta Quest head-mounted display (HMDs) consisting of a school-life karaoke experience with the hypothesis that it is possible to design the avatars with only the necessary functions and rendering costs. The school life metaverse is a relevant subject for discovering issues and problems in this type of simultaneous connection. To qualitatively evaluate the potential of a multi-person metaverse experience, this study investigated subjects where each avatar requires expressive skills. While avatar play experiences feature artistic expressions, such as dancing, playing musical instruments, and drawing, and these can be used to evaluate their operability and expressive capabilities qualitatively, the Quest’s tracking capabilities are insufficient for full-body performance and graphical art expression. Considering such hardware limitations, this study evaluated the Quest, focusing primarily on UX simplicity using AI Fusion techniques and expressiveness in instrumental scenes played by approximately four avatars. To achieve our PoC, we first researched the limitations and challenges of such applications. In Meta Quest 1 and Meta Quest 2, the benchmark for simultaneous avatar rendering was 10 for Quest 1 and 22 for the Quest 2. The appropriate interpersonal distance (IPD) should be designed with culture in mind. We assume that various avatars communicate at a maximum distance of 1–2 meters while generating collisions for individual avatars and avoiding overlap. Considering the interference in stereophonic sound, we consider that 5–11 avatars can be displayed simultaneously while maintaining quality communication. Through our standard R&D process, we assessed the production needs for our PoC and obtained multiple ways to retrieve feedback. A notable effect was observed when other users entered the same session. When users waved and called out to one another (saluted), their participation time increased by 3 minutes (i.e., they played more than one piece of music). Most users did not perceive the AI-only piano-playing avatar as human (i.e., they perceived it as a non-player character [NPC] in the game). This research reported methods for multi-user metaverse communication and its supporting technologies, such as head-mounted devices and their graphics performance, special interaction techniques, and complementary tools and the importance of PoC development, its evaluation, and its iterations. The result is remarkable for further research; these expressive technologies in a multi-user context are directly related to the quality of communication within the metaverse and the value of the user-generated content (UGC) produced there.
Language: English
Subfield: Human-Computer Interaction
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W4387412901