A topic-aware evaluation of ChatGPT’s semantic alignment with community answers using BERTScore and BERTopic
Year: 2026
Type: article
Abstract: Large language models are increasingly deployed in everyday question-answering tasks, yet their performance across diverse, real-world queries remains underexplored. This study evaluates the semantic alignment of ChatGPT’s responses with human-selected best answers in an open-domain question answering (QA) setting, using data from the Yahoo! Answers platform. Distinct from prior research that centers on domain-specific datasets, this work explores ChatGPT’s general-purpose QA performance across a wide range of topics. BERTopic is used to extract latent themes from 500 diverse full-question samples, and BERTScore metrics (precision, recall, F1) are applied to quantify semantic similarity between ChatGPT-generated responses and top-rated community answers. Results indicate that ChatGPT (GPT-3.5-turbo) achieves a strong average F1-score of 0.824, reflecting high alignment with human judgments. Topic-level analysis reveals that ChatGPT performs particularly well on factual and encyclopedia-style questions, while performance varies across more subjective or open-ended topics. This study introduces a topic-sensitive evaluation framework that enhances understanding of large language model behavior in real-world QA scenarios and supports the development of more effective and explainable conversational artificial intelligence (AI) systems.
Source: PeerJ Computer Science
Author: Mashael M. Alsulami
Institution: Taif University
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $1,395
Funder: Taif University
OpenAlex ID: W7124434995