Year: 2007
Type: dissertation
Abstract: A key requirement for developing any innovative system in a \ncomputing environment is to integrate a sufficiently friendly \ninterface with the average end user. Accurate design of such a \nuser-centered interface, however, means more than just the \nergonomics of the panels and displays. It also requires that \ndesigners precisely define what information to use and how, where, \nand when to use it. Recent advances in user-centered design of \ncomputing systems have suggested that multimodal integration can \nprovide different types and levels of intelligence to the user \ninterface. The work of this thesis aims at improving speech \nrecognition-based interfaces by making use of the visual modality \nconveyed by the movements of the lips. \n \nDesigning a good visual front end is a major part of this framework. \nFor this purpose, this work derives the optical flow fields for \nconsecutive frames of people speaking. Independent Component \nAnalysis (ICA) is then used to derive basis flow fields. The \ncoefficients of these basis fields comprise the visual features of \ninterest. It is shown that using ICA on optical flow fields yields \nbetter classification results than the traditional approaches based \non Principal Component Analysis (PCA). In fact, ICA can capture \nhigher order statistics that are needed to understand the motion of \nthe mouth. This is due to the fact that lips movement is complex in \nits nature, as it involves large image velocities, self occlusion \n(due to the appearance and disappearance of the teeth) and a lot of \nnon-rigidity. \n \nAnother issue that is of great interest to audio-visual speech \nrecognition systems designers is the integration (fusion) of the \naudio and visual information into an automatic speech recognizer. \nFor this purpose, a reliability-driven sensor fusion scheme is \ndeveloped. A statistical approach is developed to account for the \ndynamic changes in reliability. This is done in two steps. The first \nstep derives suitable statistical reliability measures for the \nindividual information streams. These measures are based on the \ndispersion of the N-best hypotheses of the individual stream \nclassifiers. The second step finds an optimal mapping between the \nreliability measures and the stream weights that maximizes the \nconditional likelihood. For this purpose, genetic algorithms are \nused. \n \nThe addressed issues are challenging problems and are substantial \nfor developing an audio-visual speech recognition framework that can \nmaximize the information gather about the words uttered and minimize \nthe impact of noise.
Source: UWSpace (University of Waterloo)
Author: Mustapha Makkook
Language: English
Topic: Speech and Audio Processing
Subfield: Signal Processing
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W2257285929