Year: 2024
Type: article
Abstract: Neural Networks (NNs) can significantly enhance perception and decision-making in resource-constrained devices like drones and wearables. However, limited resources such as memory and computing power hinder modern NN designs, leading to inaccurate predictions and delayed execution. To first ensure dependable inference, we propose modular NN structures mimicking expert decision-making. We study the effectiveness of our methodology in wireless backscatter systems under noisy channel conditions, where a modular NN is tailored for predictive power adaptation. Despite NN structure advancements, microcontroller-equipped devices still face performance barriers under extreme constraints, such as limited memory ($<$1MB) and low clock frequency ($<$300MHz). To enable more efficient use of limited resources, we propose agile offloading, which uses the patterns of feature importance identified by explainable AI to enhance the offloading efficiency. Due to the non-stationary world, NN models should also be promptly retrained d with new data, allowing it to continuously adapt to environmental dynamics and maintain its accuracy. To achieve this adaptivity, we propose a selective training scheme, where NN substructures can be freely added or skipped at runtime based on their importance with user desired computational costs. We showcase the effectiveness of our scheme on both vision and Large Language Models (LLMs). In addition to retraining upon a stationary structure, we further envision that the NN structure should be runtime expandable to accept more data modalities captured by the device. Such self-evolvability can improve the NN’s generative and reasoning capabilities in more complex tasks like autonomous navigation and human-device interaction. However, as more data modalities are incorporated, continuously enlarged models encounter scalability challenges. To mitigate training costs, we propose connecting unimodal encoders to a flexible set of last LLM blocks, training only such latent connections at runtime. We showcase its improved accuracy-compute efficiency in multimodal question-answering tasks for autonomous driving scenarios.
Author: Huang, Kai
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W6999009538
