STRUCTSENSE: A Task-Agnostic Agentic Framework for Structured Information Extraction with Human-In-The-Loop Evaluation and Benchmarking
Year: 2025
Type: preprint
Abstract: Extracting structured information from scientific literature is critical for accelerating discovery, yet Large Language Models (LLMs) often struggle in specialized domains that require expert knowledge and generalize poorly across tasks. We introduce \textsc{StructSense}, a modular, task-agnostic, open-source framework that integrates ontology-guided symbolic knowledge, agentic self-evaluative refinement, and human-in-the-loop validation for robust domain-aware extraction. We evaluate \textsc{StructSense} on three tasks of increasing semantic complexity: schema-based extraction of assessment instruments (91--100\% accuracy), metadata and resource extraction from scientific papers (86--93\% overall), and named entity recognition (NER) from neuroscience literature (58--75\% label accuracy across 8,882 entities). On two biomedical NER benchmarks (NCBI Disease and S800 Species), the system achieves $\geq$90\% relaxed recall and 62.5--85.8\% strict recall while extracting 1,000--3,600 additional entities beyond gold annotations. The local concept mapping service achieves Hits@1 of 62--82\% under strict matching and 68--86\% under semantic matching. These results across three domains demonstrate that \textsc{StructSense} generalizes across tasks while maintaining source grounding and provenance transparency.
Source: arXiv (Cornell University)
Authors: Chhetri, Tek Raj, Chen, Yibei, Trivedi, Puja, Jarecka, Dorota, Haobsh, Saif, Ray, Patrick, Ng, Lydia, Ghosh, Satrajit S.
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: green
Awards: ReproNim: A Center for Reproducible Neuroimaging Computation, DANDI: Distributed Archives for Neurophysiology Data Integration, An extensible brain knowledge base and toolset spanning modalities for multi-species data-driven cell types, BBQS AI Resource and Data Coordinating Center (BARD.CC)
OpenAlex ID: W4415344867