Work

Accelerating Successive Approximation Algorithm Via Action Elimination

Year: 2009
Type: dissertation
Abstract: This research is an effort to improve the performance of successive approximation algorithm with a prime aim of solving finite states and actions, infinite horizon, stationary, discrete and discounted\nMarkov Decision Processes (MDPs). Successive approximation is a simple and commonly used method to solve MDPs. Successive approximation often appears to be intractable for solving large scale MDPs due to its computational complexity. Action elimination, one of the techniques used to accelerate solving MDPs, reduces the\nproblem size through identifying and eliminating sub-optimal actions. In some cases successive approximation is terminated when all actions but one per state are eliminated.\n\n\n\n\nThe bounds on value functions are the key element in action elimination. New terms (action gain, action relative gain and action\ncumulative relative gain) were introduced to construct tighter bounds on the value functions and to propose an improved action\nelimination algorithm.\n\n\n\n\nWhen span semi-norm is used, we show numerically that the actual convergence of successive approximation is faster than the known theoretical rate. The absence of easy-to-compute bounds on the actual convergence rate motivated the current research to try a\nheuristic action elimination algorithm. The heuristic utilizes an estimated convergence rate in the span semi-norm to speed up action\nelimination. The algorithm demonstrated exceptional performance in terms of solution optimality and savings in computational time.\n\n\n\n\nCertain types of structured Markov processes are known to have monotone optimal policy. Two special action elimination algorithms\nare proposed in this research to accelerate successive approximation for these types of MDPs. The first algorithm uses the state space partitioning and prioritize iterate values updating in a way that maximizes temporary elimination of sub-optimal actions based on the policy monotonicity. The second algorithm is an improved version that includes permanent action elimination to improve the performance of the algorithm. The performance of the proposed algorithms are assessed and compared to that of other algorithms. The proposed algorithms demonstrated outstanding performance in\nterms of number of iterations and omputational time to converge.
Language: English
Open Access status: green
OpenAlex ID: W436268749
Cites: 87
Cited by: 3
Related to: 10