Year: 2024
Type: dissertation
Abstract: Large language models (LLMs) have become foundational resources in the natural language processing field.The capabilities of LLMs in inference, natural language understanding (NLU), and their pre-trained knowledge are widely utilized in various applications.To fully leverage these capabilities, it is essential to properly evaluate them.However, it is very challenging to evaluate LLMs accurately.For example, when evaluating an LLM's inference capabilities to infer unseen facts from seen ones, it may rely on memorized knowledge from pre-training rather than pure inference, due to the pre-training datasets being curated from web data, which can lead to contamination with related knowledge.Therefore, traditional evaluation frameworks for inference capabilities struggle to capture the pure inference capability of LLMs, independent of their pre-trained knowledge.Furthermore, when evaluating the NLU capabilities of LLMs, we should consider the stability in outputs for various input prompts.Finally, constructing evaluation datasets involves significant costs of financial, time, and human labor.Especially when focusing on multilingual datasets, there is a limitation in the availability of annotators, so most datasets are machine-translated from English.However, due to cultural differences and noise introduced by translation, these evaluations are unsuitable for language-specific evaluation.This disserta-
DOI: 10.34413/dr.02100
Author: 坂井, 優介
Institution: Nara Institute of Science and Technology
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7143759745
