Work

pxGPT: a schema-constrained multimodal large language model framework for plant phenotyping

Year: 2026

Type: article

Abstract: SUMMARY: Deep learning can scale image-based plant phenotyping but typically targets predefined traits, whereas manual phenotypic description is more flexible but difficult to scale. We present Phenotype eXplorer Generative Plant Traits (pxGPT), a Python framework that converts multi-view plant images into structured phenotype records. The workflow combines free-text description, dataset-specific schema generation with human review, and schema-constrained information extraction. We evaluated the extraction stage on three Brassica datasets using 78 973 annotations from 10 annotators who had no access to pxGPT predictions. For 38-45 qualitative traits per dataset, pxGPT-human agreement ranged from 0.572 to 0.712, compared with 0.620 to 0.739 agreement among human annotators. All 32 736 required trait fields were present; 93 values were empty, and no qualitative descriptor violations were recorded. For selected morphological structures, pooled false-positive rates were 0.5%-8.0%, while sensitivity varied across structure types in one mature-plant dataset (37.5%-100%). The curated schemas were applied unchanged with two commercial models and one locally hosted open-weight model, with differences in mean accuracy of 0.03 or less. In the evaluated Brassica datasets, these results support schema-guided extraction of visible plant traits without separate model training for each trait. Human-guided schema development offers a path to extend pxGPT to other crops and imaging conditions, with accuracy requiring validation in each new setting. AVAILABILITY AND IMPLEMENTATION: The software is available at https://github.com/YaoChengLab/pxgpt and an archival version of the source code is available on Zenodo (https://doi.org/10.5281/zenodo.22931745). Curated schemas used in this study, annotation guides, phenotype records, and human annotations are available at https://doi.org/10.5281/zenodo.22907554.

Source: Bioinformatics

DOI: 10.1093/bioinformatics/btag781