Systematic evaluation of LLM-based zero-shot model construction in chemistry and materials science: predictive performance and response reliability
Year: 2026
Type: preprint
Abstract: Purely data-driven materials exploration is often constrained by limited experimental data, motivating exploration strategies that leverage prior knowledge of the target material system encoded in large language models (LLMs). This paper describes an approach that combines LLM-derived pairwise comparisons with preference learning to construct utility-function models that rank experimental conditions with respect to a target property, without directly using measured target-property values for the system of interest either during LLM inference or model fitting. Across nine chemistry and materials science datasets and six LLMs from two providers, we evaluated predictive performance in terms of pairwise-comparison accuracy and rank agreement, and response reliability in terms of position bias, preference consistency, and abstention behavior. Predictive performance depended more strongly on the dataset than on the choice of LLM, and preference learning generally reproduced target-property rankings more accurately than direct property prediction by LLMs. Position bias and preference consistency varied across dataset–LLM combinations. Among 11 abstention configurations, the ensemble performed best overall across the risk–coverage metrics, with consistency-based and LLM-judge-based configurations also performing relatively well. Our systematic examination has characterized the predictive performance of LLM-based zero-shot modeling and the reliability of the underlying LLM output, providing insights into the potential and challenges of this approach.
Source: ChemRxiv
Authors: Shunya Maezawa, Tsuyoshi Ueno
Institution: Kwansei Gakuin University
Language: English
Subfield: Materials Chemistry
Field: Materials Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7207598790