Year: 2004
Type: book-review
Abstract: The vision of having effective algorithms score student essays should be appealing to the teacher, test publisher, and research scientist.Teachers would be freed of the burden of reading and hand-scoring maybe hundreds of student papers and consequently would be more likely to assign written questions and probe for deeper understanding.Test publishers would be able to score essays for less cost and conceivably provide higher-quality assigned grades.Research scientists, especially readers of this journal, would find this to be a fascinating area, one that merges research from multiple disciplines and having many avenues yearning for exploration.Shermis and Burstein's is the first book dedicated to the topic of automated essay scoring (AES).As such, it is destined to be the seminal work in the area.The book is composed of thirteen chapters, each written by a different set of authors.Five of the chapters provide descriptions of five different approaches and form the heart of the book.There are also four chapters on psychometric issues, two on innovations, a formal introduction, and two introductory chapters.The authors are all authorities.The five approaches are described by their developers or major advocates.As with other forms of artificial intelligence, the task of AES is to accomplish a human goal.This does not mean that the goal needs to be accomplished using the same techniques as humans use.In the case of AES, humans typically read a passage and look for certain prespecified key concepts defined in a scoring rubric.Readers call upon their content knowledge, literary experience, and language skills in evaluating the passage.The computer cannot possibly score an essay the same way.Rather, AES seeks to use the computer's special capabilities.The computer can count surface features, examine individual words and phrases, look at word order, stem, identify stop words, parse each sentence, examine sentence-to-sentence relatedness, weigh different features, identify arguments, and compare each new essay to hundreds of prescored essays.The question is whether the results are adequate.If the goal is to approximate human scores, then the answer is yes for all the approaches.Timothy Keith provides an extremely informative chapter on the predictive validity of several AES programs.The programs tend to yield impressively high correlations with the scores of human raters-generally between .70 and .90 and often between .80 and .85.Further, the correlations of AES with human raters cannot be distinguished from the correlations among human raters.As Ellis Page points out in a chapter describing Project Essay Grade (PEG), AES in a sense "passes the Turing test"-an outside observer cannot tell the difference between the machine and a human.Another way to look at the accuracy of AES is to examine the percentage of
Source: Computational Linguistics
Author: Lawrence M. Rudner
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
SDG: Quality education
Open Access status: bronze
OpenAlex ID: W1574700687
