Marru(convert): Structure Data Creation from Unstructured Text for Fine-tuning Large Language Models in Indian Languages
Year: 2024
Type: article
Abstract: This research presents an innovative methodology to convert unstructured text into a structured question-and-answer format for fine-tuning Large Language Models (LLMs) in 11 Indian languages. Addressing the scarcity of high-quality, task-specific datasets in Indian languages, the study employs a ternary quantized model based on the LLaMA-2 architecture to achieve efficient data transformation. Key contributions include the development of a parent-child text chunking system, the use of cosine similarity for enhanced data relevance, and training on 47 million tokens formatted in Alpaca style. The research further evaluates the methodology using Grice’s Maxims and AI-assisted techniques to ensure the quality and contextual accuracy of the generated datasets. This work has significant implications for improving LLM applications across domains such as education, healthcare, legal, and financial services, fostering linguistic inclusivity and technological advancement in underrepresented languages.
Authors: Mohamed Azharudeen, azharumo, Balaji Dhamodharan, Balaji
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W6912113502
FWCI:
Cites:
Cited by:
Related to: