Local response normalization. More...

#include <LocalResponseNormalization.h>

Inheritance diagram for OpenANN::LocalResponseNormalization:

Public Member Functions
	LocalResponseNormalization (OutputInfo info, double k, int n, double alpha, double beta)

virtual OutputInfo	initialize (std::vector< double * > &parameterPointers, std::vector< double * > &parameterDerivativePointers)
	Fill in the parameter pointers and parameter derivative pointers. More...

virtual void	initializeParameters ()
	Initialize the parameters. More...

virtual void	updatedParameters ()
	Generate internal parameters from externally visible parameters. More...

virtual void	forwardPropagate (Eigen::MatrixXd x, Eigen::MatrixXd &y, bool dropout, double *error=0)
	Forward propagation in this layer. More...

virtual void	backpropagate (Eigen::MatrixXd ein, Eigen::MatrixXd &eout, bool backpropToPrevious)
	Backpropagation in this layer. More...

virtual Eigen::MatrixXd &	getOutput ()
	Output after last forward propagation. More...

virtual Eigen::VectorXd	getParameters ()
	Get the current values of parameters (weights, biases, ...). More...

Public Member Functions inherited from OpenANN::Layer
virtual	~Layer ()

Detailed Description

Local response normalization.

This layer encourages the competition between neurons at the same spatial positions. It implements a form of lateral inhibition that is found in real neurons. It can be interpreted as "brightness normalization" in visual models [1].

It requires a three-dimensional input so that we can calculate the output as

$y^i_{rc} = x^i_{rc} / \left( k + \alpha \sum_{j=max(0, i-n/2)}^{min(N-1, i+n/2)} x^j_{rc} \right)^{\beta},$

where i is the index of the feature map (or filter bank), r is the row, and c the column of the neuron respectively, N is the number of feature maps and $k, n, \alpha, \beta$ are hyperparameters that have to be found with a validation set. A reasonable choice is e.g. $k=2, n=5, \alpha=10^{-4}, \beta=0.75$ [1].

[1] Krizhevsky, Alexander, Sutskever, Ilya and Hinton, Geoffrey E.: ImageNet Classification with Deep Convolutional Neural Networks, Advances in Neural Information Processing Systems 25, pp. 1106–1114, 2012.

[2] Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I. and Salakhutdinov, R. R.: Improving neural networks by preventing co-adaptation of feature detectors, 2012.

Constructor & Destructor Documentation

OpenANN::LocalResponseNormalization::LocalResponseNormalization	(	OutputInfo	info,
		double	k,
		int	n,
		double	alpha,
		double	beta
	)

Member Function Documentation

virtual void OpenANN::LocalResponseNormalization::backpropagate	(	Eigen::MatrixXd *	ein,
		Eigen::MatrixXd *&	eout,
		bool	backpropToPrevious
	)

virtual

Backpropagation in this layer.

Parameters

ein	pointer to error signal of the higher layer
eout	returns a pointer to error signal of the layer (derivative of the error with respect to the input)
backpropToPrevious	backpropagate errors to previous layers

Implements OpenANN::Layer.

virtual void OpenANN::LocalResponseNormalization::forwardPropagate	(	Eigen::MatrixXd *	x,
		Eigen::MatrixXd *&	y,
		bool	dropout,
		double *	error = `0`
	)

virtual

Forward propagation in this layer.

Parameters

x	pointer to input of the layer (with bias)
y	returns a pointer to output of the layer
dropout	enable dropout for regularization
error	error value, will be updated with regularization terms

Implements OpenANN::Layer.

virtual Eigen::MatrixXd& OpenANN::LocalResponseNormalization::getOutput ( )

virtual

Output after last forward propagation.

Returns: output

Implements OpenANN::Layer.

virtual Eigen::VectorXd OpenANN::LocalResponseNormalization::getParameters ( )

virtual

Get the current values of parameters (weights, biases, ...).

Returns: parameters

Implements OpenANN::Layer.

virtual OutputInfo OpenANN::LocalResponseNormalization::initialize	(	std::vector< double * > &	parameterPointers,
		std::vector< double * > &	parameterDerivativePointers
	)