Work

PocketLLM: A Privacy-Preserving Offline AI Assistant with On-Device LLM Inference and Retrieval-Augmented Generation on Android

Year: 2026
Type: preprint
Abstract: Cloud-based AI assistants transmit user data to remote servers, raising significant concerns around privacy, latency, and continuous internet dependency. In this paper, we present PocketLLM , a fully offline, on-device AI assistant for Android that addresses these issues by running all computations locally on the device. The system integrates a quantized large language model (LLM) deployed via llama.cpp [9], combined with a lightweight Retrieval-Augmented Generation (RAG) pipeline [5] backed by SQLite, to improve contextual understanding without any cloud dependency. PocketLLM supports natural language task execution, enabling users to make phone calls, send SMS messages, set alarms, and manage calendar events entirely on-device through an intent classification layer. The system runs the Arcee Lite 1.7B model [8] in Q8 GGUF format via a Java Native Interface (JNI) bridge on a standard 6 GB RAM smartphone, staying well within practical mobile hardware limits. Experiments carried out on 200 test queries show a mean response latency of 6.2 seconds and an intent classification accuracy of 91.5%. RAG-augmented responses outperform vanilla LLM outputs by roughly 23% on domain-specific queries, confirming the value of local knowledge retrieval. By removing any reliance on cloud infrastructure, PocketLLM keeps user data private while remaining practically usable, establishing a solid foundation for future offline intelligent assistant research.
Language: English
Open Access status: green
OpenAlex ID: W7160094446
Cites: 11
Cited by:
Related to: 10