Year: 2024
Type: dissertation
Abstract: This dissertation explores the application of zero-shot text classification, a technique for categorizing texts without annotated data in the target domain. A true zero-shot setting breaks with the conventions of the traditional supervised machine learning paradigm that relies on quantitative in-domain evaluation for optimization, performance measurement, and model selection. The dissertation summarizes existing research to build a theoretical foundation for zero-shot methods, emphasizing efficiency and transparency. It benchmarks selected approaches across various tasks and datasets to understand their general performance, strengths, and weaknesses, mirroring the model selection process. On this foundation, two case studies demonstrate the application of zero-shot text classification: The first engages with historical German stock market reports, utilizing zero-shot methods for aspect-based sentiment classification. The case study reveals that although there are qualitative differences between finetuned and zero-shot approaches, the aggregated results are not easily distinguishable, sparking a discussion about the practical implications. The second case study integrates zero-shot text classification into a civil engineering document management system, showcasing how the flexibility of zero-shot models and the omission of the training process can benefit the development of prototype software, at the cost of an unknown performance. These findings indicate that, although zero-shot text classification works for the exemplary cases, the results are not generalizable. Taking up the findings of these case studies, the dissertation discusses dilemmas and theoretical considerations that arise from omitting the in-domain evaluation of applying zero-shot text classification. It concludes by advocating a broader focus beyond traditional quantitative metrics in order to build trust in zero-shot text classification, highlighting their practical utility as well as the necessity for further exploration as these technologies evolve.:1 Introduction 1.1 Problem Context 1.2 Related Work 1.3 Research Questions & Contribution 1.4 Author’s Publications 1.5 Structure of This Work 2 Research Context 2.1 The Current State of Text Classification 2.2 Efficiency 2.3 Approaches to Addressing Data Scarcity in Machine Learning 2.4 Challenges of Recent Developments 2.5 Model Sizes and Hardware Resources 2.6 Conclusion 3 Zero-shot Text Classification 3.1 Text Classification 3.2 State-of-the-Art in Text Classification 3.3 Neural Network Approaches to Data-Efficient Text Classification 3.4 Zero-shot Text Classification 3.5 Application 3.6 Requirements for Zero-shot Models 3.7 Approaches to Transfer Zero-shot 3.7.1 Terminology 3.7.2 Similarity-based and Siamese Networks 3.7.3 Language Model Token Predictions 3.7.4 Sentence Pair Classification 3.7.5 Instruction-following Models or Dialog-based Systems 3.8 Class Name Encoding in Text Classification 3.9 Approach Selection 3.10 Conclusion 4 Model Performance Survey 4.1 Experiments 4.1.1 Datasets 4.1.2 Model Selection 4.1.3 Hypothesis Templates 4.2 Zero-shot Model Evaluation 4.3 Dataset Complexity 4.4 Conclusion 5 Case Study: Historic German Stock Market Reports 5.1 Project 5.2 Motivation 5.3 Related Work 5.4 The Corpus and Dataset - Berliner Börsenzeitung 5.4.1 Corpus 5.4.2 Sentiment Aspects 5.4.3 Annotations 5.5 Methodology 5.5.1 Evaluation Approach 5.5.2 Trained Pipeline 5.5.3 Zero-shot Pipeline 5.5.4 Dictionary Pipeline 5.5.5 Tradeoffs 5.5.6 Label Space Definitions 5.6 Evaluation - Comparison of the Pipelines on BBZ 5.6.1 Sentence-based Sentiment 5.6.2 Aspect-based Sentiment 5.6.3 Qualitative Evaluation 5.7 Discussion and Conclusion 6 Case Study: Document Management in Civil Engineering 6.1 Project 6.2 Motivation 6.3 Related Work 6.4 The Corpus and Knowledge Graph 6.4.1 Data 6.4.2 BauGraph – The Knowledge Graph 6.5 Methodology 6.5.1 Document Insertion Pipeline 6.5.2 Frontend Integration 6.6 Discussion and Conclusion 7 MLMC 7.1 How it works 7.2 Motivation 7.3 Extensions of the Framework 7.4 Other Projects 7.4.1 Product Classification 7.4.2 Democracy Monitor 7.4.3 Climate Change Adaptation Finance 7.5 Conclusion 8 Discussion: The Five Dilemmas of Zero-shot 8.1 On Evaluation 8.2 The Five Dilemmas of Zero-shot 8.2.1 Dilemma of Evaluation or Are You Working at All? 8.2.2 Dilemma of Comparison or How Do I Get the Best Model? 8.2.3 Dilemma of Annotation and Label Definition or Are We Talking about the Same Thing? 8.2.4 Dilemma of Interpretation or Am I Biased? 8.2.5 Dilemma of Unsupervised Text Classification or Do I Have to Trust You? 8.3 Trust in Zero-shot Capabilities 8.4 Conclusion 9 Conclusion 9.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 9.1.1 RQ1: Strengths and Weaknesses . . . . . . . . . . . . . . . . 139 9.1.2 RQ2: Application Studies . . . . . . . . . . . . . . . . . . . . 141 9.1.3 RQ3: Implications . . . . . . . . . . . . . . . . . . . . . . . . 143 9.2 Final Thoughts & Future Directions . . . . . . . . . . . . . . . . . . 144 References 147 A Appendix for Survey Chapter A.1 Model Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179 A.2 Task-specific Hypothesis Templates . . . . . . . . . . . . . . . . . . 180 A.3 Fractions of SotA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179 181 B Uncertainty vs. Accuracy 182 C Declaration of Authorship 185 D Declaration: Use of AI-Tools 186 E Bibliographic Data 187
Author: Borst-Graetz, Janos
Institution: Leipzig University
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W6989544408
