Year: 2024
Type: preprint
Abstract: This paper presents Llama Guard 3-1B-INT4, a compact and efficient Llama Guard model, which has been open-sourced to the community during Meta Connect 2024. We demonstrate that Llama Guard 3-1B-INT4 can be deployed on resource-constrained devices, achieving a throughput of at least 30 tokens per second and a time-to-first-token of 2.5 seconds or less on a commodity Android mobile CPU. Notably, our experiments show that Llama Guard 3-1B-INT4 attains comparable or superior safety moderation scores to its larger counterpart, Llama Guard 3-1B, despite being approximately 7 times smaller in size (440MB).
Source: arXiv (Cornell University)
Authors: Fedorov, Igor, Plawiak, Kate, Wu, Lemeng, Elgamal, Tarek, Suda, Naveen, Smith, Eric, Zhan, Hongyuan, Chi, Jianfeng, Hulovatyy, Yuriy, Patel, Kimish, Liu, Zechun, Zhao, Changsheng, Shi, Yangyang, Blankevoort, Tijmen, Pasupuleti, Mahesh, Soran, Bilge, Coudert, Zacharie Delpierre, Alao, Rachad, Krishnamoorthi, Raghuraman, Chandra, Vikas
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W4404990086