Work

A large-scale benchmark shows lightweight models can distinguish matched from mismatched problem–solution pairs across diverse STEM disciplines

Year: 2026

Type: article

Abstract: Verifying that a proposed solution truly resolves a scientific problem is central to trustworthy reasoning and retrieval. Using SCP-116K, we build 177,836 balanced problem-solution pairs (88,918 matched, 88,918 mismatched) spanning diverse STEM disciplines, and frame verification, following TRIZ/IDM, as distinguishing matched from mismatched pairs. Comparing lexical, retrieval-style, and lightweight neural models, our best model (RoBERTa + Slim ResNet, frozen sentence embeddings scored by a residual MLP) reaches an AUC of 0.966, an F1 of 0.905, and a LogLoss of 0.238. A CPU-friendly TF-IDF + Cosine + Elastic-Net baseline trails by 1.6-1.7 AUC points yet runs roughly 250× faster in about 1.5 GB of RAM, a strong efficiency-accuracy trade-off. The probabilities act as re-ranking scores over candidate solutions; we read the high ROC-AUC as pairwise discrimination and absolute accuracy as an upper bound given the synthetic negatives.

Source: Scientific Reports

DOI: 10.1038/s41598-026-60058-z

Authors: Nicolas Douard, Ahmed Samet, George Giakos, Denis Cavallucci

Institutions: Institut National des Sciences Appliquées de Strasbourg, Manhattan University

Language: English

Topic: Topic Modeling

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: gold

Estimated APC paid: $2,690

OpenAlex ID: W7166709641