Work

Improving In-context Learning Example Retrieval for Classroom Discussion Assessment with Re-ranking and Label Ratio Regulation

Year: 2025

Type: conference-paper

Abstract: Recent advancements in natural language processing, particularly large language models (LLMs), are making the automated evaluation of classroom discussions more achievable.In this work, we propose a method to improve the performance of LLMs on classroom discussion quality assessment by utilizing in-context learning (ICL) example retrieval.Specifically, we leverage example re-ranking and label ratio regulation, which forces a specific ratio of different types of examples on the ICL examples.While a standard ICL example retrieval approach shows inferior performance compared to using a predetermined set of examples, our approach improves performance in all tested dimensions.We also conducted experiments to examine the ineffectiveness of the generic ICL example retrieval approach and found that the lack of positive and hard negative examples can be a potential cause.Our analyses emphasize the importance of maintaining a balanced distribution of classes (positive, non-hard negative, and hard negative examples) in creating a good set of ICL examples, especially when we can utilize educational knowledge to identify instances of hard negative examples.

Source: Workshop on Innovative Use of NLP for Building Educational Applications

DOI: 10.18653/v1/2025.bea-1.54

Authors: Nhat Tran, Diane Litman, Benjamin Pierce, Richard Correnti, Lindsay Clare Matsumura

Institution: University of Pittsburgh

Language: English

Topic: Topic Modeling

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

SDG: Quality education

Open Access status: hybrid

Funder: University of Pittsburgh