Work

LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models

Year: 2023

Type: article

Abstract: The advent of large language models (LLMs) and their adoption by the legal community has given rise to the question: what types of legal reasoning can LLMs perform?To enable greater study of this question, we present LEGALBENCH: a collaboratively constructed legal reasoning benchmark consisting of 162 tasks covering six different types of legal reasoning.LEGALBENCH was built through an interdisciplinary process, in which we collected tasks designed and hand-crafted by legal professionals.Because these subject matter experts took a leading role in construction, tasks either measure legal reasoning capabilities that are practically useful, or measure reasoning skills that lawyers find interesting.To enable cross-disciplinary conversations about LLMs in the law, we additionally show how popular legal frameworks for describing legal reasoning-which distinguish between its many forms-correspond to LEGALBENCH tasks, thus giving lawyers and LLM developers a common vocabulary.This paper describes LEGALBENCH, presents an empirical evaluation of 20 open-source and commercial LLMs, and illustrates the types of research explorations LEGALBENCH enables. ResultsWe use LEGALBENCH to conduct a three-part study.In the first part (Section 5.2), we evaluate 20 LLMs from 11 different families, at four different size points.In the second part (Section 5.3), we show how LEGALBENCH can be used to conduct in-depth evaluations of models.To illustrate, we use LEGALBENCH to highlight similarities and differences in the performance of three popular commercial models: GPT-4, GPT-3.5, and Claude-1.In the final part (Section 5.4), we show how LEGALBENCH can support the development of law-specific LLM methods.We focus on prompting, and conduct experiments illustrating tradeoffs and challenges with regards to guiding LLMs towards certain tasks.Ultimately, our study here serves to illustrate the types of analyses that LEGALBENCH enables, and highlight potential directions for future work.We summarize the findings of our study here, and provide more details in the Appendix. Setup ModelsWe study 20 LLMs from 11 different families.These are: GPT-3.5 [14] (text-davinci-003) and GPT-4 from OpenAI [96]; Claude-1 (v1.3) from Anthropic [3]; Incite-Instruct-7B, Incite-Base-7B, and Incite-Instruct-3B from Together [34, 133]; OPT-2.7B,OPT-6.7B, and OPT-13B from Meta [154]; Falcon-7B-Instruct from TII [2, 103]; MPT-7B-8k-Instruct from MosaicML [129]; Vicuna-7B-16k and Vicuna-13B-16k from LMSYS [26] ; Flan-T5-XL (3B parameters) and Flan-T5-XXL (11B parameters) from Google [31]; LLaMA-2-7B, and LLaMA-2-13B from Meta [134]; WizardLM-13B [150]; and BLOOM-3b and BLOOM-7B from BigScience [116].All inference was performed on two-GPU GCP 40GB A100s, using the Manifest library [97].HuggingFace links for each model are provided in the Appendix.

Source: eYLS (Yale Law School)