Work

Scalable Evaluation of Retrieval-Augmented Generation Systems for Educational Question Answering

Year: 2026
Type: article
Abstract: AI tutors powered by Retrieval-Augmented Generation (RAG) are increasingly used in education, but their reliability depends on retrieving appropriate instructional content. This study quantifies RAG performance in an educational setting using benchmarks of varying complexity derived from a biotechnology curriculum. We began with a baseline RAG system evaluated on a manually constructed question–answer benchmark, then introduced automated benchmark generation to scale evaluation across RAG configurations and identify the most impactful design choices. We further evaluated the best-performing configuration on textbook-section benchmarks labeled by Bloom's Taxonomy and Webb's Depth of Knowledge. Results show that architectural choices such as retrieval strategy and chunking were associated with larger performance differences than parameter-level adjustments, with Milvus hybrid retrieval achieving higher success rates than Pinecone dense retrieval in this benchmark. Full-textbook retrieval also improved performance on some higher-difficulty questions compared to section-limited search. Together, these findings underscore the importance of systematically evaluating retrieval reliability for AI tutors in complex biotechnology domains.
Language: English
Open Access status: green
OpenAlex ID: W7235407741
FWCI:
Cites:
Cited by:
Related to: 10