Evaluating the Performance of Direct Preference Optimization in News Article Title Generation : A Comparative Analysis with Supervised Fine-Tuning
Year: 2024
Type: dissertation
Abstract: Typically, Large Language Models (LLMs) are adapted to a specific task using large amounts of labeled training data. This process has been found to teach the LLM undesirable behavior, an issue that’s been effectively solved by a recent training method incorporating human preference, called Direct Preference Optimization (DPO). However, this method has largely been limited to instruction-based LLMs. This thesis seeks to evaluate the performance of Supervised Fine-Tuning (SFT) and DPO and several of its variations in a new, non-instruction-based context: news article title generation. However, due to the subjective nature of the task, simple objective measures are not enough, and the large number of DPO variations and hyperparameter values is infeasible to evaluate using only human evaluators. This thesis solves these issues by using a multi-step pipeline consisting of both objective and subjective metrics for model selection and evaluation. Using this method, this thesis finds that the most performant DPO variation as determined by objective measures did not perform significantly differently from the SFT variant of the same base model in either LLM or human evaluations. This indicates that either DPO was unable to learn the patterns of successful article titles better than SFT, or that the method used was unable to detect a difference that would have been found in real-world A/B testing. These results show that in some tasks no quality difference between DPO and SFT is detectable by human experts nor by an LLM. Hopefully, these conclusions will be helpful to future research in the area, and perhaps these conclusions could be confirmed or rejected using proper A/B testing.
Author: Dettner Källander, Hugo
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7018961719