Group-aware validation sharply reduces the apparent accuracy of extraction-category classification in public cannabis laboratory data
Year: 2026
Type: preprint
Abstract: Published models that classify cannabis products from cannabinoid and terpene profiles report near-perfect accuracy: above 95% for resin type, 100% for chemovar class. These figures come from random cross-validation or Kennard-Stone partitioning, which hold out no plant, producer or laboratory. We quantify the cost on a task with explicit group structure. On 33,229 public laboratory-result rows for cannabis concentrates that name a producer, nearly all from Nevada, each with a source-declared extraction category (non-solvent against solvent-based), a gradient-boosted classifier scores 78.5% balanced accuracy and a ROC AUC of 0.89 under random five-fold cross-validation. Holding out the producer leaves 54.0% and 0.62. A signal survives: the pooled held-out AUC is 0.61, with a producer-level bootstrap interval of 0.535 to 0.698. But the reported model's pooled balanced accuracy does not separate from chance (51.6%, interval 47.3 to 56.6); it keeps about 30% of its above-chance AUC, and an unweighted logistic regression 41 to 53%. A designed experiment shows the same direction: with the random split averaged over 200 seeds, 162 HPLC-assayed samples fall from 54.3% to 38.9% under leave-one-variety-out with gradient boosting and from 51.8% to 46.3% with logistic regression, against 33.3% chance. In the market data the same profiles identify the producer at 67.1% against a 14.4% baseline and the laboratory at 92.3% against 40.3%. We conclude that the declared extraction category is only weakly recoverable from these analyte panels for an unseen producer, and that a random split overstates it by 8 to 26 points of balanced accuracy depending on the model and subset. We did not rerun the published studies.
Source: ChemRxiv
Author: Arthur Gonçalves Breguez
Language: English
Subfield: Analytical Chemistry
Field: Chemistry
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7221163875