From Visual Saliency to Video Behaviour Understanding
Year: 2007
Type: dissertation
Abstract: In a world of ever increasing amounts of video data, we are forced to abandon traditional \nmethods of scene interpretation by fully manual means. Under such circumstances, some form \nof automation is highly desirable but this can be a very open ended issue with high complexity. \nDealing with such large amounts of data is a non-trivial task that requires efficient selective \nextraction of parts of a scene which have the potential to develop a higher semantic meaning, \nalone, or in combination with others. In particular, the types of video data that are in \nneed of automated analysis tend to be outdoor scenes with high levels of activity generated \nfrom either foreground or background. Such dynamic scenes add considerable complexity \nto the problem since we cannot rely on motion energy alone to detect regions of interest. \nFurthermore, the behaviour of these regions of motion can differ greatly, while still being \nhighly dependent, both spatially and temporally on the movement of other objects within \nthe scene. Modelling these dependencies, whilst eliminating as much redundancy from the \nfeature extraction process as possible are the challenges addressed by this thesis. \nIn the first half, finding the right mechanism to extract and represent meaningful features \nfrom dynamic scenes with no prior knowledge is investigated. Meaningful or salient information \nis treated as the parts of a scene that stand out or seem unusual or interesting to \nus. The novelty of the work is that it is able to select salient scales in both space and time \nin which a particular spatio-temporal volume is considered interesting relative to the rest of \nthe scene. By quantifying the temporal saliency values of regions of motion, it is possible to \nconsider their importance in terms of both the long and short-term. Variations in entropy \nover spatio-temporal scales are used to select a context dependent measure of the local scene \ndynamics. A method of quantifying temporal saliency is devised based on the variation of \nthe entropy of the intensity distribution in a spatio-temporal volume over incraeasing scales. \nEntropy is used over traditional filter methods since the stability or predictability of the intensity \ndistribution over scales of a local spatio-temporal region can be defined more robustly \nrelative to the context of its neighbourhood, even for regions exhibiting high intensity variation \ndue to being extremely textured. Results show that it is possible to extract both locally \nsalient features as well as globally salient temporal features from contrasting scenerios. \nIn the second part of the thesis, focus will shift towards binding these spatio-temporally \nsalient features together so that some semantic meaning can be inferred from their interaction. \nInteraction in this sense, refers to any form of temporally correlated behaviour between \nany salient regions of motion in a scene. Feature binding as a mechanism for interactive \nbehaviour understanding is particularly important if we consider that regions of interest may \nnot be treated as particularly significant individually, but represent much more semantically \nwhen considered in combination. Temporally correlated behaviour is identified and classified \nusing accumulated co-occurrences of salient features at two levels. Firstly, co-occurrences are \naccumulated for spatio-temporally proximate salient features to form a local representation. \nThen, at the next level, the co-occurrence of these locally spatio-temporally bound features \nare accumulated again in order to discover unusual behaviour in the scene. The novelty of \nthis work is that there are no assumptions made about whether interacting regions should be \nspatially proximate. Furthermore, no prior knowledge of the scene topology is used. Results \nshow that it is possible to detect unusual interactions between regions of motion, which can \nvisually infer higher levels of semantics. \nIn the final part of the thesis, a more specific investigation of human behaviour is addressed \nthrough classification and detection of interactions between 2 human subjects. Here, further \nmodifications are made to the feature extraction process in order to quantify the spatiotemporal \nsaliency of a region of motion. These features are then grouped to find the people \nin the scene. Then, a loose pose distribution model is extracted for each person for finding \nsalient correlations between poses of two interacting people using canonical correlation \nanalysis. These canonical factors can be formed into trajectories and used for classification. \nLevenshtein distance is then used to categorise the features. The novelty of the work is that \nthe interactions do not have to be spatially connected or proximate for them to be recognised. \nFurthermore, the data used is outdoors and cluttered with non-stationary background. Results \nshow that co-occurrence techniques have the potential to provide a more generalised, \ncompact, and meaningful representation of dynamic interactive scene behaviour.
Author: Hayley Shi Wen Hung
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W2545326849