EST: A Morphology-Guided English-to-Sanskrit Tokenizer for Compact NLP Representations
Year: 2025
Type: article
Abstract: We present EST, a tokenizer that maps English text into linguistically valid Sanskrit tokens, achieving 55.3 % average token reduction (peak 82 %) while preserving 95.2 % context. A 33 425-entry Paninian lexicon and a dual-path architecture ensure 100 % coverage with zero token loss. The system is released under an open licence for immediate reuse.
Author: Patil, Sumedh
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7108440684
FWCI:
Cites:
Cited by:
Related to: