Work

Using k-Shot Prompting with Large k for the Automated Scoring of a German Written Elicited Imitation Test

Year: 2026

Type: conference-paper

Abstract: This paper explores the application of a Large Language Model (LLM) using k-shot prompting with large k for automatically scoring a German Written Elicited Imitation Test (WEIT), a test for assessing literacy-dependent procedural knowledge in German as a foreign language.In this test, test-takers are briefly presented with written sentences which they then have to reproduce in writing as accurately as possible.The responses are scored on an ordinal scale which differentiates between different types of errors (e.g.lexical vs. grammatical).We find that with increasing k (in a range from 1 to 700) accuracy increases significantly but it also depends on the drawn sample and varies across different runs of the same prompt.Overall, the kshot setting which relies on in-context learning without being provided with the scoring rubric outperforms a baseline where only the scoring rubric is provided to the model.However, the LLM does not outperform previous results based on rule-based or BERT-based models.

Source: Workshop on Innovative Use of NLP for Building Educational Applications

DOI: 10.18653/v1/2026.bea-1.20

Authors: Malte Sternik, Ronja Laarmann-Quante, Anastasia Drackert

Language: English

Topic: Natural Language Processing Techniques

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

SDG: Quality education

Open Access status: hybrid

Funder: Deutsche Forschungsgemeinschaft

Award: Zeit Lücken zu schließen: eine multimethodische Studie zur Rolle der Zeit im Konstrukt computergestützter CTests in drei Sprachen