Efficient Training of Feed-Forward Neural Networks

Abstract

Since the discovery of the back-propagation method, many modified and new algorithms have been proposed for training of feed-forward neural networks. The problem with slow convergence rate has, however, not been solved when the training is on large scale problems. There is still a need for more efficient algorithms. This Ph.D. thesis describes different approaches to improve convergence. The main results of the thesis is the development of the Scaled Conjugate Gradient Algorithm and the stochastic version of this algorithm. Other important results are the development of methods that can derive and use Hessian information in an efficient way. The main part of this thesis is the 5 papers presented in appendices A-E. Chapters 1-6 give an overview of learning in feed-forward neural networks, put these papers in perspective and present the most important results. The conclusion of this thesis is:

* Conjugate gradient algorithms are very suitable for training of feed-forward networks.

* Use of second order information by calculations on the Hessian matrix can be used to improve convergence.