Year: 2026
Type: conference-paper
Abstract: LLMs are increasingly able to answer complex questions about enterprise-scale document collections.But evaluation is hard: companies don't want to share internal communications, and synthetic datasets have been overly simple.We present CORPORATEBENCH (CB), a human-validated multi-task Q&A benchmark whose scale approaches the conditions LLMs encounter in corporate communication networks, with evaluation corpora surpassing 230,000 documents.CB evaluates LLMs across two dimensions (information extraction and knowledge base querying) through four synthetically generated firms ranging from 12 to 10,000 employees.Each corpus is sampled from a temporally evolving knowledge base describing a consistent world, guaranteeing crossdocument logical consistency even across hundreds of thousands of documents.We evaluate five LLMs on CB, revealing increasingly poor performance as input size approaches realistic scales.CB provides LLM developers a metric for corporate communication reasoning, filling a crucial gap in the benchmarking ecosystem.
Institution: Cornell University
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: bronze
OpenAlex ID: W7204581906