When Generative Artificial Intelligence Meets Extended Reality: Enabling Scalable and Natural Interactions
Year: 2025
Type: article
Abstract: Extended Reality (XR), including virtual, augmented, and mixed reality, provides immersive and interactive experiences across diverse applications, from VR-based education to AR-based assistance and MR-based training. However, widespread XR adoption remains limited due to two key challenges: 1) the high cost and complexity of authoring 3D content, especially for large-scale environments or complex interactions; and 2) the steep learning curve associated with non-intuitive interaction methods like handheld controllers or scripted gestures. Generative AI (GenAI) presents a promising solution by enabling intuitive, language-driven interaction and automating content generation. Leveraging vision-language models and diffusion-based generation, GenAI can interpret ambiguous instructions, understand physical scenes, and generate or manipulate 3D content, significantly lowering barriers to XR adoption. This paper explores the integration of XR and GenAI through three concrete use cases, showing how they address key obstacles in scalability and natural interaction, and identifying technical challenges that must be resolved to enable broader adoption.
Source: IEEE Internet Computing
Authors: Mingyu Zhu, Jiangong Chen, Bin Li
Institution: Pennsylvania State University
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Awards: CAREER: Wireless Collaborative Mixed Reality Networking: Foundations and Algorithms for Joint Communication, Computation, and Learning, EAGER: TaskDCL: Collaborative Research: IMPACT: Interactive Mixed-Reality-Based Platform for AI-Driven Adaptive and Collaborative Task Training Environments, G557427397
OpenAlex ID: W7123359105