OmniSense: A Unified Framework for Multimodal Human-State, Device-Motion, Interaction-Behavior, and Environmental Awareness During Generative AI Inference
Year: 2026
Type: preprint
Abstract: Contemporary generative artificial intelligence (AI) systems typically operate through a request–generation–delivery paradigm: a user submits an instruction, the system generates an output, and the completed result is presented for inspection. Although streaming interfaces may expose intermediate tokens, frames, or audio segments, the underlying generation process is not necessarily responsive to continuously evolving information about the user's physical behavior, device movement, interaction patterns, or surrounding environment. Consequently, an AI system may continue generating an output even when the user's needs, attention, physical activity, interaction context, or environmental circumstances have changed substantially during generation. This paper proposes OmniSense, a unified framework for multimodal human-state and environmental awareness during generative AI inference. The framework extends beyond conventional facial, speech, and textual signals to encompass bodily movements, hand and finger gestures, wearable-device measurements, smartphone and smart-glasses motion, keyboard and mouse activity, touchscreen interactions, scrolling and navigation patterns, device orientation, proximity, contextual environmental observations, and other appropriately authorized signals. These heterogeneous observations are temporally aligned, uncertainty-calibrated, and transformed into a continuously updated contextual state representation. A generation-control mechanism then determines whether and how the remaining, not-yet-committed portion of an output should be adapted. The proposed architecture supports dynamic modification of generation objectives, content priorities, structural organization, visual composition, temporal characteristics, and computational allocation across text, image, video, audio, and multimodal generation. Unlike post-generation editing or conventional feedback-driven regeneration, OmniSense explicitly treats the ongoing generation process as an adaptive control problem. A relevant existing patent, US12518461B1, Real-time adaptable interactive AI persona, is examined as a closely related prior disclosure. Its claims expressly address real-time modification during ongoing generation based on facial, voice, and textual emotional information. Its description also mentions input hardware such as keyboards, touchpads, and mice. Accordingly, this paper does not claim that real-time user-responsive generation or the use of interaction hardware is unprecedented. Instead, it identifies a narrower research contribution: a general-purpose, cross-modal architecture that unifies continuous sensing of human movement, device motion, interaction behavior, and environmental context with uncertainty-aware control of unfinished generative outputs across different generation tasks. The framework is conceptual and testable; no experimental superiority is claimed.
Author: Zhang, Jincheng
Language: English
Topic: Speech and dialogue systems
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225806073
Cites:
Cited by:
Related to: