Year: 2025
Type: preprint
Abstract: With the improving semantic understanding capability of Large Language Models (LLMs), they exhibit a greater awareness and alignment with human values, but this comes at the cost of transparency. Although promising results are achieved via experimental analysis, an in-depth understanding of the LLM's internal workings is unavoidable to comprehend the reasoning behind the re-ranking, which provides end users with an explanation that enables them to make an informed decision. Moreover, in newly developed systems with limited user engagement and insufficient ranking data, accurately re-ranking content remains a significant challenge. While various training methods affect the training of LLMs and generate inference, our analysis has found that some training methods exhibit better explainability than others, implying that an accurate semantic understanding has not been learned through all training methods; instead, abstract knowledge has been gained to optimize evaluation, which raises questions about the true reliability of LLMs. Therefore, in this work, we analyze how different training methods affect the semantic understanding of the re-ranking task in LLMs and investigate whether these models can generate more informed textual reasoning to overcome the challenges of transparency or LLMs and limited training data. To analyze the LLMs for re-ranking tasks, we utilize a relatively small ranking dataset from the environment and the Earth science domain to re-rank retrieved content. Furthermore, we also analyze the explainable information to see if the re-ranking can be reasoned using explainability.
Source: arXiv (Cornell University)
Authors: Islam, Nafis Tanveer, Zhao, Zhiming
Institution: University of Amsterdam
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Awards: LTER-LIFE: a research infrastructure to develop Digital Twins of ecosystems in a changing world, G4733186887, O.S.C.A.R.S. - Open Science Clusters’ Action for Research and Society, ENVironmental Research Infrastructures delivering an open access Hub and NEXT-level interdisciplinary research framework providing services for advancing science and society, European Virtual Institute for Research Software Excellence, A federated European FAIR and Open Research Ecosystem for oceans, seas, coastal and inland waters
OpenAlex ID: W7078545545
Cites:
Cited by:
Related to: