Systematic Evaluation for Large Language Model Integration: Benchmarking Metrics, Compute Footprint, and Latency Impact in LLMs
Year: 2025
Type: article
Abstract: As Large Language Models (LLMs) become central to modern Natural Language Processing (NLP), the need for rigorous benchmarking—across latency, compute efficiency, cost, and accuracy—has never been more critical. This paper presents a robust comparative analysis of leading LLMs including OpenAI’s GPT series, Anthropic’s Claude, Google’s Gemini, Meta’s LLaMA, DeepSeek, Nova, and local deployment alternatives like Ollama. Through standardized metrics (e.g., MMLU, HumanEval, GSM8K, DROP, MGSM, CMath, CMMLU, CLUEWSC, C-Eval), architectural profiling (dense vs. mixture-of-experts), fine-tuning and inference considerations, and real-world deployment benchmarks, we provide practitioners with the insights needed for intelligent model onboarding. Our analysis further includes inference latency, cost-per-token, and energy efficiency implications—ultimately equipping developers and researchers to align model selection with task complexity and production constraints. This research provides engineers and decision-makers with a crucial framework for selecting and deploying Large Language Models, balancing cutting-edge accuracy with real-world constraints like latency, cost, and compute resources. By benchmarking leading models including GPT-4, Claude 3.5, Gemini 1.5, and DeepSeek across diverse metrics, our findings directly inform the efficient integration of LLMs into applications ranging from real-time chatbots and code generation tools to complex RAG systems, enabling more effective and economically viable AI solutions.
Author: Anvesh reddy minukuri
Institution: JPMorgan Chase & Co (United States)
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W4410306339