Fairbank, Michael and Prokhorov, Danil and Alonso, Eduardo (2014) 'Clipping in Neurocontrol by Adaptive Dynamic Programming.' IEEE Transactions on Neural Networks and Learning Systems, 25 (10). 1909 - 1920. ISSN 2162-237X

Preview

Text CLIPPING-RCO.pdf - Accepted Version Download (387kB) | Preview

Abstract

In adaptive dynamic programming, neurocontrol, and reinforcement learning, the objective is for an agent to learn to choose actions so as to minimize a total cost function. In this paper, we show that when discretized time is used to model the motion of the agent, it can be very important to do clipping on the motion of the agent in the final time step of the trajectory. By clipping, we mean that the final time step of the trajectory is to be truncated such that the agent stops exactly at the first terminal state reached, and no distance further. We demonstrate that when clipping is omitted, learning performance can fail to reach the optimum, and when clipping is done properly, learning performance can improve significantly. The clipping problem we describe affects algorithms that use explicit derivatives of the model functions of the environment to calculate a learning gradient. These include backpropagation through time for control and methods based on dual heuristic programming. However, the clipping problem does not significantly affect methods based on heuristic dynamic programming, temporal differences learning, or policy-gradient learning algorithms.

Item Type:	Article
Uncontrolled Keywords:	Backpropagation through time (BPTT); clipping; dual heuristic programming (DHP); neurocontrol; value-gradient learning
Subjects:	Q Science > QA Mathematics > QA75 Electronic computers. Computer science
Divisions:	Faculty of Science and Health > Computer Science and Electronic Engineering, School of
Depositing User:	Jim Jamieson
Date Deposited:	17 Nov 2015 12:44
Last Modified:	12 Mar 2020 14:15
URI:	http://repository.essex.ac.uk/id/eprint/15484

Actions (login required)

View Item