Work

Local Moment Agreement Does Not Preserve Decision Exponents in Coupled Q-Learning

Year: 2026
Type: preprint
Abstract: We study wrong greedy decisions made from the last iterate of constant-step Q-learning at stationarity. Bias, variance and a Gaussian approximation describe the distribution near the optimal values. We ask how much they tell us about rare decision errors. For one learner, the frozen-policy linearization gives the exact error exponent in every finite MDP with deterministic transitions, noisy rewards and synchronous sampling, under the assumptions stated below. This property can fail for the Richardson-Romberg (RR) combination of two learners. Both learners use the true Bellman max and share the same samples. In a bounded one-state example, the real pair and its linearization agree in every fixed polynomial moment up to errors smaller than any power of the step size. Yet the ratio of their RR error probabilities has an exponent above 0.07315. A three-state cycle with independent reward noises gives a bound above 0.0538. Both bounds also hold against a real ordinary learner with the same leading variance. We prove a stationary variational formula and use it to study the effect of the discount factor. With Gaussian noise and step ratio 𝑐 > 1, the RR exponent divided by the exponent of the variance-matched learner tends to 𝑐/(𝑐 + 1) in the one-state model as the discount factor tends to one. On every cycle of fixed length, the limsup is at most this value. For 𝑐 = 2, the loss is one third in the scalar model and at least one third in the cycle limit. We also give a small-discount expansion with a quadratic remainder. These results concern unaveraged outputs and do not challenge existing bias or averaging guarantees for RR.
Author: Zheng, Yaxin
Language: English
Open Access status: green
OpenAlex ID: W7226009372
Cites:
Cited by:
Related to: 10