Research indexPreprint
Addressing Function Approximation Error in Actor-Critic Methods
Fujimoto, van Hoof, Meger
arXiv (Cornell University)26 Feb 2018Open access
NobleIDark:/48914/w1/20260525/8BC19B64DOI10.48550/arxiv.1802.09477
Addressing Function Approximation Error in Actor-Critic Methods
Fujimoto, van Hoof, Meger · arXiv (Cornell University) · 2018
NobleIDark:/48914/w1/20260525/8BC19B64DOI10.48550/arxiv.1802.09477
Abstract
In value-based reinforcement learning methods such as deep Q-learning, function approximation errors are known to lead to overestimated value estimates and suboptimal policies. We show that this problem persists in an actor-critic setting and propose novel mechanisms to minimize its effects on both the actor and the critic. Our algorithm builds on Double Q-learning, by taking the minimum value between a pair of critics to limit overestimation. We draw the connection between target networks and overestimation bias, and suggest delaying policy updates to reduce per-update error and further improve performance. We evaluate our method on the suite of OpenAI gym tasks, outperforming the state of the art in every environment tested.