Work

Efficient Algorithms for the Uniform Tokenization Problem

Year: 2025

Type: article

Abstract: Tokenization (also known as scanning or lexing) is a computational task that has applications in the lexical analysis of programs during compilation and in data extraction and analysis for unstructured or semistructured data (e.g., data represented using the JSON and CSV data formats). We propose two algorithms for the tokenization problem that have linear time complexity (in the length of the input text) without using large amounts of memory. We also show that an optimized version of one of these algorithms performs well compared to prior approaches on practical tokenization workloads.

Source: Proceedings of the ACM on Programming Languages

DOI: 10.1145/3720498

Authors: Angela W. Li, Konstantinos Mamouras

Institution: Rice University

Language: English

Topic: Algorithms and Data Compression

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: diamond

Funder: National Science Foundation

Award: CAREER: Programming Abstractions and Formal Reasoning for IoT Application Development