Year: 2022
Type: preprint
Abstract: It is well-known that for sparse linear bandits, when ignoring the dependency on sparsity which is much smaller than the ambient dimension, the worst-case minimax regret is $\widetildeΘ\left(\sqrt{dT}\right)$ where $d$ is the ambient dimension and $T$ is the number of rounds. On the other hand, in the benign setting where there is no noise and the action set is the unit sphere, one can use divide-and-conquer to achieve $\widetilde{\mathcal O}(1)$ regret, which is (nearly) independent of $d$ and $T$. In this paper, we present the first variance-aware regret guarantee for sparse linear bandits: $\widetilde{\mathcal O}\left(\sqrt{d\sum_{t=1}^T σ_t^2} + 1\right)$, where $σ_t^2$ is the variance of the noise at the $t$-th round. This bound naturally interpolates the regret bounds for the worst-case constant-variance regime (i.e., $σ_t \equiv Ω(1)$) and the benign deterministic regimes (i.e., $σ_t \equiv 0$). To achieve this variance-aware regret guarantee, we develop a general framework that converts any variance-aware linear bandit algorithm to a variance-aware algorithm for sparse linear bandits in a "black-box" manner. Specifically, we take two recent algorithms as black boxes to illustrate that the claimed bounds indeed hold, where the first algorithm can handle unknown-variance cases and the second one is more efficient.
Source: arXiv (Cornell University)
Authors: Dai, Yan, Wang, Ruosong, Du, Simon S.
Institutions: Tsinghua University, University of Washington
Language: English
Field: Decision Sciences
Domain: Social Sciences
Open Access status: green
Awards: IIS:RI Theoretical Foundations of Reinforcement Learning: From Tabula Rasa to Function Approximation, CAREER: Toward a Foundation of Over-Parameterization, Collaborative Research: SCALE MoDL: Adaptivity of Deep Neural Networks, Collaborative Research: CIF: Medium: MoDL:Toward a Mathematical Foundation of Deep Reinforcement Learning, AI Institute: Institute for Foundations of Machine Learning
OpenAlex ID: W4281691852