Intelligent Context-Aware Screen Assistance System for Real-Time Productivity Enhancement
Year: 2026
Type: article
Abstract: Fixing a problem on a computer usually means leaving the task. The user copies an error message, searches the web, and then tries to apply what they find. This paper describes a screen assistance system that removes most of that detour. It watches the screen, reads the visible text with optical character recognition (OCR), identifies interface elements, and builds a structured description of what the user is doing. When it finds a problem, it rates the severity and asks a large language model (LLM) for a fix. The answer appears as an issue card with the root cause, numbered steps, a command that can be copied, and a confidence value. The assistant runs in a React dashboard and in a floating Windows overlay, and it also provides a prompt library, a local knowledge base, voice mode, and a searchable history of past sessions. Suggestions come from a large language model (LLM), and the user's feedback on each one is stored so that later answers can improve. What sets this work apart is the combination: continuous screen understanding, automatic detection of problems with severity levels, and an LLM explanation of each problem, all in one desktop tool. The multi-utility assistant that inspired this project [4] concentrates on commands and utilities rather than on watching what the user is doing. We also treat privacy as a starting point. Nothing is captured until the user agrees, capture can be paused at any time, and stored data is encrypted. The prototype was tested at unit, integration, functional, and system levels. In a worked example, it recognized a missing-module error in a terminal and proposed the correct repair, which suggests that proactive, screen-aware help is practical on an ordinary desktop.
Language: English
Subfield: Human-Computer Interaction
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225589474