Year: 2023
Type: preprint
Abstract: Semi-structured tables are ubiquitous. There has been a variety of tasks that aim to automatically interpret, augment, and query tables. Current methods often require pretraining on tables or special model architecture design, are restricted to specific table types, or have simplifying assumptions about tables and tasks. This paper makes the first step towards developing open-source large language models (LLMs) as generalists for a diversity of table-based tasks. Towards that end, we construct TableInstruct, a new dataset with a variety of realistic tables and tasks, for instruction tuning and evaluating LLMs. We further develop the first open-source generalist model for tables, TableLlama, by fine-tuning Llama 2 (7B) with LongLoRA to address the long context challenge. We experiment under both in-domain setting and out-of-domain setting. On 7 out of 8 in-domain tasks, TableLlama achieves comparable or better performance than the SOTA for each task, despite the latter often has task-specific design. On 6 out-of-domain datasets, it achieves 5-44 absolute point gains compared with the base model, showing that training on TableInstruct enhances the model's generalizability. We open-source our dataset and trained model to boost future work on developing open generalist models for tables.
Source: arXiv (Cornell University)
Authors: Zhang, Tianshu, Yue, Xiang, Li, Yifei, Sun, Huan
Institution: The Ohio State University
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Awards: CAREER: Towards Interactive and Transparent Question Answering with Applications in the Clinical Domain, AI Institute for Intelligent CyberInfrastructure with Computational Learning in the Environment (ICICLE), III: Small: Towards Resolving Ad-hoc Concept Queries with Table Answers via Multi-source Data Mining, G6671297155
OpenAlex ID: W4388748516