LLMSR@XLLM25: Integrating Reasoning Prompt Strategies with Structural Prompt Formats for Enhanced Logical Inference
Year: 2025
Type: conference-paper
Abstract: This paper illustrates our team system approach in XLLM-ACL 2025 Task-III: LLM for Structural Reasoning (LLM-SR), aiming to solve both Task: Question parsing and CoT parsing.The process of extracting statements and evidence is similar to Discourse Parsing.Correct extraction of statements or evidence from the COT is crucial at the outset.Next, the pairwise relationship between a specific statement and its corresponding evidence is assessed (a statement should be followed by its related evidence from the CoT).Both semantic and lexical similarity are used to evaluate the accuracy of statements and evidence predictions.Finally, once a statement-evidence pair is correctly extracted, it is evaluated to determine whether the evidence can logically deduce the statement.To tackle Question Parsing and CoT Parsing, we implement and investigate various solutions, including (1) applying different structural prompt formats like JSON, Markdown, or XML.(2) utilising various prompt techniques: Few-shot, Chain of thought, and Multi-hop prompting.(3) Taking advantage of Natural Language Inference (NLI) model for the Statement Verification step.Our best official result is a 243.047 mean score for test phases A and B, and finally, we rank 7th on the final leaderboard.
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
OpenAlex ID: W4412944186