Year: 2023
Type: article
Abstract: What happened?Who? When?Where?Why?What will happen next?are the fundamental questions asked to comprehend the overwhelming amount of information.Answers to these questions are the core knowledge communicated through multiple forms of information, regardless of whether presented as text, images, videos, audio, or other modalities.To obtain such knowledge from multimodal data, this dissertation focuses on Multimodal Information Extraction (IE), and propose Event-Centric Multimodal Knowledge Acquisition to evolve traditional Entity-centric Single-modality knowledge into Eventcentric Multi-modality knowledge.Traditional entity-centric approaches to consuming multimodal information focus on concrete concepts (such as objects, object types, physical relations, e.g., a person in a car ), while this dissertation endows machines to understand complex abstract semantic structures that are difficult to ground into image regions but are essential knowledge (such as events and semantic roles of objects, e.g., driver, passenger, passerby, salesperson).It is able to consolidate complex semantic structures of multiple modalities, providing a major benefit over recent research advances in single-modality (text-only or vision-only) knowledge.Such a transformation poses significant challenges in terms of understanding multimodal semantic structures (such as semantic roles) and temporal dynamics (such as future participants and their roles):• Understanding Multimodal Semantic Structures to answer What happened?, Who?, Where?, and When? (Knowledge Extraction): Due to the structural nature and lack of anchoring in a specific image region, abstract semantic structures are difficult to synthesize between text and vision modalities through general large-scale pretraining.We introduce complex event semantic structures into vision-language pretraining (CLIP-Event), and propose a zero-shot cross-modal transfer of semantic understanding abilities from language to vision, which resolves the poor portability issue of IE and supports Zero-shot Multimodal Event Extraction (M 2 E 2 ) for the first time.We also release an open-source Multimodal IE system GAIA to serve as an off-the-shelf tool for the research community.• Understanding Temporal Dynamics to answer What will happen next?, Who will participant?and Why? (Knowledge Reasoning): The significance of capturing temporal dynamics has led to recent advances in script knowledge learning, however, Over the past five years, I have had an invaluable journey.I had the unique opportunity to witness this transformative era of natural language processing (NLP).When I embarked on my PhD in late 2018, I was still taking classes about statistical NLP approaches, just when large pre-trained models like BERT started to gain attraction, gradually growing in scale and ambition, now dominating most NLP applications we see today.I experienced this rapid development firsthand and was both thrilled, and occasionally panicked, to be a part of this wave.I have been fortunate enough to meet many, many people who have supported me along my journey in various ways, and I am very grateful for their help.If I am allowed to use one word to summarize what I learned during my PhD studies, that would be "courage".Upon starting my PhD, I had limited understanding of what is good research.At that time, I considered research to be keeping up with the trend, and really cared about the numbers in result tables.However, having had the incredible fortune of working alongside these esteemed mentors, I started to learn the meaning of "courage" -to boldly face challenging tasks and to fearlessly pose new questions.As I first delved into multimodal deep semantic understanding, I felt lonely due to the limited existing work in the field, but I gradually learn to enjoy such loneliness when working on novel challenges.The unwavering passion of these mentors inspired me, and I began to recognize the essence of a true researcher and the research I wanted to pursue.I am deeply indebted to these mentors for their wisdom and guidance.My greatest thanks go to my advisor Heng Ji.She has been an extraordinary role model for me, leaving an indelible mark on my life and shaping the foundation of my research philosophy.She taught me how to appreciate and refine the art of research, and encouraged me to be a nail rather than a hammer.She showed me how to enjoy research, and most importantly, how to be a brave leader to tackle novel challenges.Beyond research, she worked so hard to promote female leadership, and her goal to send a female PI troop to funding agencies have profoundly influenced me.She is so much more than my academic advisor, she is a trusted friend with whom I can discuss anything.She helped me see value in myself that I hadn't seen before.I already find myself missing her.I would like to thank Shih-Fu Chang for being on my thesis committee and for providing extensive guidance throughout my PhD studies.When I began working on multimodal deep semantic understanding, I had no prior knowledge of computer vision.Shih-Fu offered inv valuable guidance and extensive support in this area.I learned the beauty of interdisciplinary research from him.He is an extremely kind, patient, and supportive mentor.I am grateful to have Kyunghyun Cho on my thesis committee.I was fortunate to work with one of the most brilliant minds in our field.He possesses a high-level vision of the field and can always grasp the essence of the problems deeply.He is so patient and every discussions were filled with insightful ideas.I would also like to extend my heartfelt thanks to Jiawei Han, a titan in the fields of data and text mining.His book of Data Mining: Concepts and Techniques opened doors to the world of research for me as an undergraduate.He exemplifies the qualities of diligence, academic rigor, and compassion, who is indeed an inspirational role model for me.It is also my great honor to have Chengxiang Zhai on my thesis committee.I am constantly intrigued by his zeal for science, as well as his wide-ranging interests, at all times remaining curious about new research challenges.He always reminds me to look at my research from a broader perspective and encourages me to discover my true passion in the next few decades.I am grateful for his guidance.
Author: Li, Manling
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7220823226