Levenberg-Marquardt Algorithm. More...

#include <LMA.h>

Inheritance diagram for OpenANN::LMA:

Public Member Functions
	LMA ()

virtual	~LMA ()

virtual void	setOptimizable (Optimizable &opt)
	Pass the objective function. More...

virtual void	setStopCriteria (const StoppingCriteria &stop)
	Pass the stop criteria. More...

virtual void	optimize ()
	Optimize until the optimization meets the stop criteria. More...

virtual bool	step ()
	Execute one optimization step. More...

virtual Eigen::VectorXd	result ()
	Determine the best result. More...

virtual std::string	name ()
	Get the name of the optimization algorithm. More...

Public Member Functions inherited from OpenANN::Optimizer
virtual	~Optimizer ()

Detailed Description

Levenberg-Marquardt Algorithm.

This algorithm works especially well for least squares optimization. The optimization will stop if one of the following stopping criteria is satisfied:

stop.minimalSearchSpaceStep
$|E^{t+1}-E^{t}| \leq$ stop.minimalValueDifferences $\cdot max\{|E^{t+1}|,|E^{t}|,1\}$
stop.maximalIterations

The optimization is cubic in each iteration, i. e. the complexity is $ O(L^3) $ , where L is the number of parameters to optimize. In addition, the jacobian matrix is required for this algorithm, that is the required space is in $ O(L N) $ , where N is the size of the training set. However, LMA requires only a few iterations to converge. Thus, it works very well up to a few thousand parameters and a few thousand training examples. In summary, the requirements are:

sum of squared error (SSE)
maximum of a few thousand parameters
maximum of a few thousand training examples

[1] Kenneth Levenberg: A Method for the Solution of Certain Problems in Least Squares, Quarterly of Applied Mathematics 2, pp. 164-168, 1944.

[2] Donald Marquardt: An Algorithm for Least-Squares Estimation of Nonlinear Parameters, Journal of the Society for Industrial and Applied Mathematics 11 (2), pp. 431-441, 1963.