Fully connected layer. More...

#include <FullyConnected.h>

Inheritance diagram for OpenANN::FullyConnected:

Public Member Functions
	FullyConnected (OutputInfo info, int J, bool bias, ActivationFunction act, double stdDev, Regularization regularization)

virtual OutputInfo	initialize (std::vector< double * > &parameterPointers, std::vector< double * > &parameterDerivativePointers)
	Fill in the parameter pointers and parameter derivative pointers. More...

virtual void	initializeParameters ()
	Initialize the parameters. More...

virtual void	updatedParameters ()
	Generate internal parameters from externally visible parameters. More...

virtual void	forwardPropagate (Eigen::MatrixXd x, Eigen::MatrixXd &y, bool dropout, double *error=0)
	Forward propagation in this layer. More...

virtual void	backpropagate (Eigen::MatrixXd ein, Eigen::MatrixXd &eout, bool backpropToPrevious)
	Backpropagation in this layer. More...

virtual Eigen::MatrixXd &	getOutput ()
	Output after last forward propagation. More...

virtual Eigen::VectorXd	getParameters ()
	Get the current values of parameters (weights, biases, ...). More...

Public Member Functions inherited from OpenANN::Layer
virtual	~Layer ()

Protected Attributes
int	I

int	J

bool	bias

ActivationFunction	act

double	stdDev

Eigen::MatrixXd	W

Eigen::MatrixXd	Wd

Eigen::VectorXd	b

Eigen::VectorXd	bd

Eigen::MatrixXd *	x

Eigen::MatrixXd	a

Eigen::MatrixXd	y

Eigen::MatrixXd	yd

Eigen::MatrixXd	deltas

Eigen::MatrixXd	e

Regularization	regularization

Detailed Description

Fully connected layer.

Each neuron in the previous layer is taken as input for each neuron of this layer. Forward propagation is usually done by $a = W \cdot x + b, y = g(a)$ , where $ a $ is the activation vector, $ y $ is the output, $ g $ a typically nonlinear activation function that operates on a vector, $ x $ is the input of the layer, $ W $ is a weight matrix and $ b $ is a bias vector.

Actually, we have a faster implementation that computes the forward and backward pass for N instances in parallel. Suppose we have an input matrix $ X $ with $ N $ rows and $ I $ columns, i.e. each row contains an input vector $ x $ . The forward propagation is implemented in two steps:

$ A = X W^T, Y = g(A) $

and the backpropagation is

$\Delta = g'(A) * \frac{\partial E}{\partial Y}, \frac{\partial E}{\partial X} = \Delta W, \frac{\partial E}{\partial W} = \Delta^T X, \frac{\partial E}{\partial b} = \sum_n \Delta_n$

Neural networks with one fully connected hidden layer and a nonlinear activation function are universal function approximators, i. e. with a sufficient number of nodes any function can be approximated with arbitrary precision. However, in practice the number of nodes could be very large and overfitting is a problem. Therefore it is sometimes better to add more hidden layers. Note that this could cause another problem: the gradients vanish in the lower layers such that these cannot be trained properly. If you want to apply a complex neural network to tasks like image recognition you could instead try Convolutional layers and pooling layers (MaxPooling, Subsampling) in the lower layers. These can be trained surprisingly well in deep architectures.

Supports the following regularization types:

L1 penalty
L2 penalty
Maximum of squared norm of the incoming weight vector

[1] Kurt Hornik, Maxwell B. Stinchcombe and Halbert White: Multilayer feedforward networks are universal approximators, Neural Networks 2 (5), pp. 359-366, 1989.

Constructor & Destructor Documentation

OpenANN::FullyConnected::FullyConnected	(	OutputInfo	info,
		int	J,
		bool	bias,
		ActivationFunction	act,
		double	stdDev,
		Regularization	regularization
	)

Member Function Documentation

virtual void OpenANN::FullyConnected::backpropagate	(	Eigen::MatrixXd *	ein,
		Eigen::MatrixXd *&	eout,
		bool	backpropToPrevious
	)

virtual

Backpropagation in this layer.

Parameters

ein	pointer to error signal of the higher layer
eout	returns a pointer to error signal of the layer (derivative of the error with respect to the input)
backpropToPrevious	backpropagate errors to previous layers

Implements OpenANN::Layer.

virtual void OpenANN::FullyConnected::forwardPropagate	(	Eigen::MatrixXd *	x,
		Eigen::MatrixXd *&	y,
		bool	dropout,
		double *	error = `0`
	)

virtual

Forward propagation in this layer.

Parameters

x	pointer to input of the layer (with bias)
y	returns a pointer to output of the layer
dropout	enable dropout for regularization
error	error value, will be updated with regularization terms

Implements OpenANN::Layer.

virtual Eigen::MatrixXd& OpenANN::FullyConnected::getOutput ( )

virtual

Output after last forward propagation.

Returns: output

Implements OpenANN::Layer.

virtual Eigen::VectorXd OpenANN::FullyConnected::getParameters ( )

virtual

Get the current values of parameters (weights, biases, ...).

Returns: parameters

Implements OpenANN::Layer.

virtual OutputInfo OpenANN::FullyConnected::initialize	(	std::vector< double * > &	parameterPointers,
		std::vector< double * > &	parameterDerivativePointers
	)

virtual

Fill in the parameter pointers and parameter derivative pointers.

Parameters

parameterPointers	pointers to parameters
parameterDerivativePointers	pointers to derivatives of parameters

Returns: information about the output of the layer

Implements OpenANN::Layer.

virtual void OpenANN::FullyConnected::initializeParameters ( )

virtual

Initialize the parameters.

This is usually called before each optimization.

Implements OpenANN::Layer.

virtual void OpenANN::FullyConnected::updatedParameters ( )

virtual

Generate internal parameters from externally visible parameters.

This is usually called after each parameter update.

Implements OpenANN::Layer.

Member Data Documentation

Eigen::MatrixXd OpenANN::FullyConnected::a

protected

ActivationFunction OpenANN::FullyConnected::act

protected

Eigen::VectorXd OpenANN::FullyConnected::b

protected

Eigen::VectorXd OpenANN::FullyConnected::bd

protected

bool OpenANN::FullyConnected::bias

protected

Eigen::MatrixXd OpenANN::FullyConnected::deltas

protected

Eigen::MatrixXd OpenANN::FullyConnected::e

protected

int OpenANN::FullyConnected::I

protected

int OpenANN::FullyConnected::J

protected

Regularization OpenANN::FullyConnected::regularization

protected

double OpenANN::FullyConnected::stdDev

protected

Eigen::MatrixXd OpenANN::FullyConnected::W

protected

Eigen::MatrixXd OpenANN::FullyConnected::Wd

protected

Eigen::MatrixXd* OpenANN::FullyConnected::x

protected

Eigen::MatrixXd OpenANN::FullyConnected::y

protected

Eigen::MatrixXd OpenANN::FullyConnected::yd

protected

The documentation for this class was generated from the following file:

OpenANN/layers/FullyConnected.h

Public Member Functions

Protected Attributes

Detailed Description

Constructor & Destructor Documentation

Member Function Documentation

Member Data Documentation