Year: 2025
Type: conference-paper
Abstract: Automated scientific idea generation systems have made remarkable progress, yet the automatic evaluation of idea novelty remains a critical and underexplored challenge.Manual evaluation of novelty through literature review is labor-intensive, prone to error due to subjectivity, and impractical at scale.To address these issues, we propose the Idea Novelty Checker, an LLM-based retrieval-augmented generation (RAG) framework that leverages a two-stage retrieve-then-rerank approach.The Idea Novelty Checker first collects a broad set of relevant papers using keyword and snippet-based retrieval, then refines this collection through embedding-based filtering followed by facetbased LLM re-ranking.It incorporates expertlabeled examples to guide the system in comparing papers for novelty evaluation and in generating literature-grounded reasoning.Our extensive experiments demonstrate that our novelty checker achieves approximately 13% higher agreement than existing approaches.Ablation studies further showcases the importance of the facet-based re-ranker in identifying the most relevant literature for novelty evaluation.Figure 6: contd.TextGrad Prompt Optimisation.
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W4412944169