Work

System As A Collaborator (SAAC): a Framework for Modeling, Capturing and Augmenting Collaborative Activities in Extended Reality

Year: 2025

Type: conference-paper

Abstract: When humans collaborate on a shared task, they use a myriad of verbal, para-verbal and non-verbal cues to achieve this end. Modern Artificial Intelligence sensing techniques such as Social Signals Processing (SSP) allow the characterization of users’ activities in Augmented or Virtual Environments through the analysis of heterogeneous multimodal data sources, such as interaction actions, gaze direction, avatars’ positions, and speech analysis. Therefore, this enables realtime assessment of team processes, including communication or team situation awareness. In return, it can provide context-specific feedback and information to augment team capabilities. However, implementing this vision remains a technical challenge, particularly in gathering real-time data sources in multi-user collaborative scenarios. This paper presents a framework for assessing and augmenting team collaboration with Extended Reality Environments (XRE). We propose a multimodal architecture to capture users’ activities and augment the team capabilities by displaying system-generated context-specific collaborative cues. Our proposed vision, System As A Collaborator (SAAC), frames the XRE as a direct actor embedded in the collaborative activity, improving team members’ capabilities with collaborative augmentations instead of being merely the space where collaboration occurs. We demonstrate the feasibility of our approach and system architecture through use cases of experimental XRE where our software architecture collects multimodal, heterogeneous, and multi-user interaction, behavioral and physiological data, and generates direct, reactive cues, and higher-level context-specific cues, augmenting team collaboration.

Source: 2022 IEEE 18th International Conference on Automation Science and Engineering (CASE)

DOI: 10.1109/case58245.2025.11164117

Authors: Aurélien Lechappé, Aurélien Milliat, Alexandre Kabil, Mathieu Chollet, Cédric Dumas

Institutions: Centre National de la Recherche Scientifique, Laboratoire des Sciences du Numérique de Nantes, IMT Atlantique, University of Glasgow, Laboratoire Interdisciplinaire des Sciences du Numérique

Language: English

Topic: Virtual Reality Applications and Impacts

Subfield: Human-Computer Interaction

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

Funder: Agence Nationale de la Recherche

Awards: Modelling the Induction of Social Stress in Immersive Virtual rEality Simulations, CoMixS : Collaborative design in MIXed-presence interactive Spaces, Controlling Self and the Other under Emotional Tension, Resources Restrictions and Stress