llmsays: One-Line LLM Inference with Automatic Prompt-Tier Routing and Multi-Provider Failover
Year: 2026
Type: article
Abstract: llmsays is an open-source Python library that reduces large language model (LLM) inference to a single function call. It introduces a two-stage hybrid routing mechanism that classifies an input prompt into one of fourcomplexity tiers — small, medium, large, and extra_large — using a lightweight sentence-transformer model,and subsequently dispatches the query to the most appropriate provider-specific model. The library supportsfive commercial inference providers (Groq, NVIDIA NIM, OpenRouter, Fireworks AI, and Baseten) with automatic latency-aware failover, eliminating manual provider management for researchers and practitioners alike.An optional multiprocessing mode enables parallel provider queries, returning the first successful response forlatency-critical applications.
Authors: Chand, Anik, Adhikary, Abhiraj, Bhowmick, Rudra
Institution: Haldia Institute of Technology
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7147438328
FWCI:
Cites:
Cited by:
Related to: