Portable system for monitoring emotional state and concentration in interviews based on multi-modal fusion of visual and audio data
Year: 2026
Type: article
Abstract: <<S.M.H.Mousavi_Patent_Claims>> Patent Title: Portable system for monitoring emotional state and concentration in interviews based on multi-modal fusion of visual and audio data on a single-board computer platform. Claims What is claimed is: Claim 1) A non-contact device for monitoring and analyzing human behavioral, emotional, and cognitive states, comprising: a processing unit based on a Raspberry Pi board, a color image capture camera, an audio input, a wireless mini touch keyboard and a touch stylus, an LED-based lighting module, and a touch display; configured such that upon receiving power from an external power supply and activating the switch integrated into the power supply, the operating system boots and the Python-based behavioral analysis software is launched manually or semi-automatically; wherein the device is capable of capturing and processing video and audio data simultaneously, in a non-contact and real-time manner, at a frame rate ranging from 11 to 15 frames per second. Claim 2) According to Claim 1, the device is configured to automatically detect faces, determine and stabilize the target facial region without physical contact, and extract visual data including eye movements, blinking, gaze direction, mouth activity, head movements, body movements, and respiratory patterns. Claim 3) According to Claim 1, speech audio data is received via the audio input (microphone), and acoustic speech features are extracted without storing raw audio content, and are processed synchronously with the visual features. Claim 4) According to Claims 2 and 3, the extracted visual and audio features are combined through multi-channel data fusion to calculate high-level behavioral indices, including stress, focus, self-confidence, and derived indices related to mental adaptability and resilience. Claim 5) According to Claim 1, the processing results consist of features and numerical indices, which are displayed numerically and trend-wise via the touch display, such that no raw audio or video data containing personally identifiable information is stored. Claim 6) According to Claim 1, the device is designed such that the extracted data and numerical indices are automatically stored, and upon completion of the monitoring process, system shutdown is executed via a software Shut Down command followed by a complete power cutoff using the power supply switch.
Author: Mousavi, Seyed Muhammad Hossein
Language: English
Topic: Emotion and Mood Recognition
Subfield: Experimental and Cognitive Psychology
Field: Psychology
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7224774151