Work

Förstärkt inlärning offline för hantering av radioresurser i radioaccessnät

Year: 2024
Type: dissertation
Abstract: Recent work has shown the potential of applying reinforcement learning (RL) to radio resource management (RRM) in radio access networks (RAN) thanks to their ability to learn complex parameter relationships by interaction with an unknown environment. However, real-world RAN deployments are challenging as the training phase may temporarily degrade the system’s performance up to, potentially, triggering link failure procedures. To address such issues, we consider a particular family of RL algorithms, namely offline RL methods, where agents are trained on previously collected datasets of transitions. We investigate the application of offline RL methods for link adaptation (LA) - a crucial functionality of RRM. In particular, we design and evaluate multiple offline RL algorithms for LA, including batch-constrained Q-learning (BCQ), conservative Q-learning (CQL), behavioral cloning (BC) and decision transformer (DT), and compare their performance against two baselines: an outer-loop link adaptation (OLLA) algorithm, as commonly adopted in today’s RAN systems; and an online off-policy deep Q-network (DQN) algorithm for LA. Since DT, differently from other offline RL methods, treats RL as a sequence modeling problem, we propose modifications to the algorithm design and the problem formulation according to the use-case. Our results show that all considered methods reach comparable performance to the DQN policy (with also a slight improvement) and better performance than OLLA. However, contrarily to traditional applications of transformer models, which benefit from long input sequences for generative predictions, we observed that the stochastic nature of the radio systems prevents DT from properly conditioning the generation of optimal actions at inference time on long input sequences, whereas performing well with shorter sequences. The thesis also includes a preliminary study conducted on Gym environments providing insights into the influence of the training datasets on performances of offline RL policies.
Language: English
Open Access status: green
OpenAlex ID: W7000405614
Cites: 42
Cited by:
Related to: 10