Year: 2015
Type: article
Abstract: Document-level school lesson quality classification based on German transcriptsAnalyzing large-bodies of audiovisual information with respect to discoursepragmatic categories is a time-consuming, manual activity, yet of growing importance in a wide variety of domains.Given the transcription of the audiovisual recordings, we propose to model the task of assigning discoursepragmatic categories as supervised machine learning task.By analyzing the effects of a wide variety of feature classes, we can trace back the discoursepragmatic ratings to low-level language phenomena and better understand their dependency.The major contribution of this article is thus a rich feature set to analyze the relationship between the language and the discoursepragmatic categories assigned to an analyzed audiovisual unit.As one particular application of our methodology, we focus on modelling the quality of lessons according to a set of discourse-pragmatic dimensions.We examine multiple lesson quality dimensions relevant for educational researchers, e.g. to which extent teachers provide objective feedback, encourage cooperation and pursue thinking pathways of students.Using the transcripts of real classroom interactions recorded in Germany and Switzerland, we identify a wide range of lexical, stylistic and discourse-pragmatic phenomena, which affect the perception of lesson quality, and we interpret our findings together with the educational experts.Our results show that especially features focusing on discourse and cognitive processes are beneficial for this novel classification task, and that this task has a high potential for automated assistance.Band 30 (1) -99-124 Flekova, Sousa, Mieskes, Gurevych trace back the discourse-pragmatic ratings to low-level language phenomena and better understand their dependency. JLCL 2015 -As one particular application of our methodology, we focus on modelling the quality of lessons according to a set of discourse-pragmatic dimensions.Educational researchers extensively analyze the interaction between teachers and students in all age groups in order to find components of teaching effectiveness.A commonly employed method is based on videography, in which acoustical and visual elements of lessons are recorded.Researchers analyze these recordings and assess the quality of interaction both between the teacher and the students, and among the students, on a variety of levels, in order to design teacher trainings and educational interventions.This assessment is a very tedious process, as multiple trained experts have to evaluate the quality independently.During this process, each annotator has to go over the complete material several times in order to assign quality scores on the various dimensions and levels of depth relevant for educational researchers.For details on the procedure see e.g.Rakoczy (2006).Currently, this task is carried out completely manually, which does not allow for scaling such studies to a wider scope.Supporting it with automatic methods would help to reduce this impediment.We show that our feature set can reveal important insights into the nature of language associated with the studies of quality dimensions of the lessons.These findings can be utilized in a number of ways: i) to automate the task of analyzing the quality dimensions manually, and thus scale the technology to monitor huge amounts of data, and ii) to provide feedback to educational specialists about the lessons and help to conceptualize the appropriate interventions.The feature set itself can be re-used in the future for similar tasks.In close cooperation with educational researchers, we processed a data set of transcripts of German mathematics lessons and created multiple machine learning models to classify the lessons on a range of quality dimensions.Thanks to the manual expert annotations, provided by educational researchers, we were able to train and test each model on a highly reliable gold standard.We make both the data 1 and the software 2 available to the research community.Our key contribution is two-fold: Firstly, we determine which aspects of the verbal behaviour expressed in the transcripts (such as sentiment polarity or discourse structure) are predictive for the ratings of lesson quality dimensions (such as constructive feedback, student cooperation and reasoning path development), and we offer interpretation of our findings.Secondly, by modeling the problem as a text classification task with NLP features, we also demonstrate that this educational research task has a high potential for automation.Our initial results show that especially features related to e.g.discourse, sentiment and cognitive processes allow for good lesson classification within the quality dimensions studied here.The data set of transcripts is publicly available.To our 1 https://www.ukp.tu-darmstadt.de/index.php?id=11716 2 https://github.com/UKPLab/pythagoras-Kindly keep in mind that the aim of publishing the experimental code is mainly to provide a reference for feature implementation details rather than to distribute a fully documented and tested piece of software.
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
SDG: Quality education
Open Access status: hybrid
Awards: G348198870, G6467963393, G7578079922
OpenAlex ID: W2584218128
