Year: 2024
Type: preprint
Abstract: String processing, which mainly involves the analysis and manipulation of strings, is a fundamental component of modern computing. Despite the significant advancements of large language models (LLMs) in various natural language processing (NLP) tasks, their capability in string processing remains underexplored and underdeveloped. To bridge this gap, we present a comprehensive study of LLMs' string processing capability. In particular, we first propose StringLLM, a method to construct datasets for benchmarking string processing capability of LLMs. We use StringLLM to build a series of datasets, referred to as StringBench. It encompasses a wide range of string processing tasks, allowing us to systematically evaluate LLMs' performance in this area. Our evaluations indicate that LLMs struggle with accurately processing strings compared to humans. To uncover the underlying reasons for this limitation, we conduct an in-depth analysis and subsequently propose an effective approach that significantly enhances LLMs' string processing capability via fine-tuning. This work provides a foundation for future research to understand LLMs' string processing capability. Our code and data are available at https://github.com/wxl-lxw/StringLLM.
Source: arXiv (Cornell University)
Authors: Wang, Xilong, Fu, Hao, Wang, Jindong, Gong, Neil Zhenqiang
Institution: Duke University
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Funder: National Science Foundation
Awards: CAREER: Graph-Based Security Analytics: New Algorithms, Robustness under Adversarial Settings, and Robustness Enhancements, Collaborative Research: SaTC: CORE: Small: Securing Recommender Systems against Data Poisoning Attacks, AI Institute for Edge Computing Leveraging Next Generation Networks (Athena)
OpenAlex ID: W4403853715