Work

Learning multisensory representations from unlabeled videos

Year: 2020
Type: dissertation
Abstract: Recently, several researches have been carried forward about self-supervised learning, with\nthe goal of remedying the necessity of large amount of labeled data that characterises\nsupervised deep learning approaches. Indeed, although supervised learning methods may\nachieve impressive performance, labeling huge amount of data is often impracticable since\nit may require expert knowledge and/or data may be scarcely available. Through the use\nof an auxiliary task, called pretext task, a self-supervised model can be trained without\nthe need of manual annotations using as pseudo-labels information obtained directly from\nthe data. Typically, the pretext task is defined to give a capability to the network useful\nfor the final task, also called downstream task, to which it should be correlated.\nThis project aims to develop a multimodal self-supervised method, that learns an\nembedded space for audio and visual video data to provide a reasonable feature representations of specific actions in the EPIC Kitchen Dataset. Using the concatenation of\ntwo different proxy tasks, mainly audio-video synchronization and audio-video alignment,\na deep learning model can obtain strong temporal reasoning capacity able to describe\nan action in a multimodal space used help to represent the main features of the action.\nTo validate the quality of the learnt representation, a fine-grained cross-retrieval task for\naudio and videos is performed on the features extracted from the entire dataset. Moreover\nsome classifiers are trained on the same features to test the capacity of the embedded\nspace in term of action recognition.\nThis thesis was developed in collaboration with IRII - Institut de Robòtica i Informàtica\nIndustrial in Barcelona, Spain, as final project of the master in Automation Engineering,\nheld at University of Padova, Italy
Author: Toson, Giorgio
Institution: University of Padua
Language: English
Open Access status: green
OpenAlex ID: W7048514691
Cites: 53
Cited by:
Related to: 10