Towards a Metrological Validation of PFAS Property Predictions by In-Context LLMs Across Four Architectures
Year: 2026
Type: preprint
Abstract: In recent years, the use of large-language models (LLMs) as prediction tools has become increasingly popular. Despite this popularity, the metrological validation of LLMs in the study of chemical problems has been somehow neglected to date. To illustrate this need in chemistry, we introduce a minimal, metrologically grounded framework for validating in-context learning (ICL)-based prediction in LLMs on molecular physicochemical properties with reliable estimation of uncertainty. Our framework integrates a multicycle testing protocol (Mc = 50 replicate cycles per model-property cell) with a six-metric bundle: central tendency (mean MAE over replicate cycles), reproducibility (interquartile range), regression performance (RMSE, and R2), information-theoretical lower bound on the amount of captured signal (𝐼" in nats), and ordinal 2 performance (accuracy against the median in the dataset), plus three cross-property and crossmodel comparison metrics: NRMSE, Welch’s z-test, and Scaling Law Fit exponents. We use this framework to validate four LLMs, two dense (8B and 70B parameters; 1B = 109) and two mixture-of-experts (MoE; 17B active parameters with 109B and 400B in total), for predicting eight per- and polyfluoroalkyl substance (PFAS) properties: boiling point (BP), melting point (MP), vapor pressure (VP), Henry’s law constant (HLC), octanol-water partition coefficient (LogKow), aqueous solubility (LogS), acid dissociation constant (pKa), and critical micelle concentration (LogCMC). In this benchmark, the frontier MoE model (Maverick 400B parameters) achieves a mean validation NRMSE of 14.4 %, and the mid-scale MoE model (Scout, 109B parameters) outperforms our larger dense model (Llama-3.3-70B) despite having 4.1× fewer active parameters. This suggests that MoE routed capacity can substitute for activeparameter scale in chemical applications of ICL. When compared to published QSPR and DFT baselines, LLM-ICL beats these approaches on three of eight directly comparable properties (pKa by 43 %, LogKow by 31 %, HLC by 10 %) and trails behind on five (BP, MP, LogS, VP by 12 % to 75 %, and LogCMC by a factor of approx. 3.3), despite using approximately 32 incontext examples (32 for seven properties, 20 for HLC) versus the training sets comprising 102 to 105 molecules used in these approaches. We also discuss a validation-set anomaly in the case of LogKow, where our mid-scale dense model Llama-3.3-70B outperforms the frontier MoE model on all four validation metrics, possibly due to better coverage of the pretraining data. Permutation (shuffled-label) and information-access blinding controls show that genuine structure-response signal is present, but that the native-prompt accuracy depends substantially on pretrained, property-specific knowledge and familiar target scales. Overall, our results suggest a two-fold conclusion on the competitiveness of LLM-ICL: it succeeds in predicting chemical
Source: ChemRxiv
Author: Carlos Gonzalez
Language: English
Subfield: Materials Chemistry
Field: Materials Science
Domain: Physical Sciences
Keywords: large language models, in-context learning, metrological validation, PFAS, physicochemical property prediction, mixture of experts, uncertainty estimation, boiling point, melting point, vapor pressure, Henry's law constant, octanol-water partition coefficient, water solubility, acid dissociation constants, critical micelle concentration, QSPR
Open Access status: green
OpenAlex ID: W7236605714
