Restricted Boltzmann Machine. More...

#include <RBM.h>

Inheritance diagram for OpenANN::RBM:

Public Member Functions
	RBM (int D, int H, int cdN=1, double stdDev=0.01, bool backprop=true, Regularization regularization=Regularization())
	Construct RBM. More...

virtual	~RBM ()

virtual Eigen::VectorXd	operator() (const Eigen::VectorXd &x)
	Make a prediction. More...

virtual Eigen::MatrixXd	operator() (const Eigen::MatrixXd &X)
	Make predictions. More...

virtual bool	providesInitialization ()
	Check if the object knows how to initialize its parameters. More...

virtual void	initialize ()
	Initialize the optimizable parameters. More...

virtual unsigned int	examples ()
	Request number of training examples. More...

virtual unsigned int	dimension ()
	Request the number of optimizable parameters. More...

virtual void	setParameters (const Eigen::VectorXd &parameters)
	Set new parameters. More...

virtual const Eigen::VectorXd &	currentParameters ()
	Request the current parameters. More...

virtual double	error ()
	Compute error on training set. More...

virtual double	error (unsigned int n)

virtual bool	providesGradient ()
	Check if the object provides a gradient of the error function with respect to its parameters. More...

virtual Eigen::VectorXd	gradient ()
	Compute gradient of the error function with respect to the parameters. More...

virtual Eigen::VectorXd	gradient (unsigned int n)

virtual void	errorGradient (std::vector< int >::const_iterator startN, std::vector< int >::const_iterator endN, double &value, Eigen::VectorXd &grad)
	Calculates the accumulated gradient and error of given training examples. More...

virtual void	forwardPropagate (Eigen::MatrixXd x, Eigen::MatrixXd &y, bool dropout, double *error=0)
	Forward propagation in this layer. More...

virtual void	backpropagate (Eigen::MatrixXd ein, Eigen::MatrixXd &eout, bool backpropToPrevious)
	Backpropagation in this layer. More...

virtual Eigen::MatrixXd &	getOutput ()
	Output after last forward propagation. More...

virtual Eigen::VectorXd	getParameters ()
	Get the current values of parameters (weights, biases, ...). More...

virtual OutputInfo	initialize (std::vector< double * > &parameterPointers, std::vector< double * > &parameterDerivativePointers)
	Fill in the parameter pointers and parameter derivative pointers. More...

virtual void	initializeParameters ()
	Initialize the parameters. More...

virtual void	updatedParameters ()
	Generate internal parameters from externally visible parameters. More...

int	visibleUnits ()
	Get number of visible units. More...

int	hiddenUnits ()
	Get number of hidden units. More...

const Eigen::MatrixXd &	getWeights ()
	Get the current weight matrix. More...

const Eigen::MatrixXd &	getVisibleProbs ()
	Get probabilities of visible units. More...

const Eigen::MatrixXd &	getVisibleSample ()
	Get sample from the distribution of the visible units. More...

Eigen::MatrixXd	reconstructProb (int n, int steps)
	Get the probability of the n-th training example after some sampling steps. More...

void	sampleHgivenV ()
	Sample hidden units given visible units. More...

void	sampleVgivenH ()
	Sample visible units given hidden units. More...

Public Member Functions inherited from OpenANN::Learner
	Learner ()

virtual	~Learner ()

virtual Learner &	trainingSet (Eigen::MatrixXd &input, Eigen::MatrixXd &output)
	Set training set. More...

virtual Learner &	trainingSet (DataSet &trainingSet)
	Set training set. More...

virtual Learner &	removeTrainingSet ()
	Remove the training set from the learner. More...

virtual Learner &	validationSet (Eigen::MatrixXd &input, Eigen::MatrixXd &output)
	Set validation set. More...

virtual Learner &	validationSet (DataSet &validationSet)
	Set validation set. More...

virtual Learner &	removeValidationSet ()
	Remove the validation set from the learner. More...

Public Member Functions inherited from OpenANN::Optimizable
virtual	~Optimizable ()

virtual void	finishedIteration ()
	This callback is called after each optimization algorithm iteration. More...

virtual double	error (unsigned n)
	Compute error of a given training example. More...

virtual Eigen::VectorXd	gradient (unsigned n)
	Compute gradient of a given training example. More...

virtual void	errorGradient (int n, double &value, Eigen::VectorXd &grad)
	Calculates the function value and gradient of a training example. More...

virtual void	errorGradient (double &value, Eigen::VectorXd &grad)
	Calculates the function value and gradient of all training examples. More...

virtual Eigen::VectorXd	error (std::vector< int >::const_iterator startN, std::vector< int >::const_iterator endN)
	Calculates the errors of given training examples. More...

virtual Eigen::VectorXd	gradient (std::vector< int >::const_iterator startN, std::vector< int >::const_iterator endN)
	Calculates the accumulated gradient of given training examples. More...

Public Member Functions inherited from OpenANN::Layer
virtual	~Layer ()

Additional Inherited Members
Protected Attributes inherited from OpenANN::Learner
DataSet *	trainSet

DataSet *	validSet

bool	deleteTrainSet

bool	deleteValidSet

int	N

Detailed Description

Restricted Boltzmann Machine.

RBMs have been originally invented by Paul Smolensky in 1986 [1] and since contrastive divergence [2] can be used to calculate an approximation of a gradient, we can efficiently train RBMs. RBMs are usually used to learn features unsupervised. However, they can be stacked and we can use them to initialize deep autoencoders for dimensionality reduction or feedforward networks for classification. Standard RBMs assume that the data is binary (at least approximately, i.e. the values have to be within [0, 1]).

Deep networks are usually difficult to train because the required learning rate in the first layer is usually much higher than in the upper layers. This problem can be solved by initializing the first layers with RBMs, which was the major breakthrouh in deep learning. There are also other ways to make deep learning work, e.g. CNNs (weight sharing), ReLUs, maxout, etc.

Supports the following regularization types:

L1 penalty
L2 penalty

[1] Smolensky, Paul: Information Processing in Dynamical Systems: Foundations of Harmony Theory, MIT Press, 1986, pp. 194-281.

[2] Hinton, Geoffrey E.: Training Products of Experts by Minimizing Contrastive Divergence, Technical Report, University College London, 2000.

Constructor & Destructor Documentation

OpenANN::RBM::RBM	(	int	D,
		int	H,
		int	cdN = `1`,
		double	stdDev = `0.01`,
		bool	backprop = `true`,
		Regularization	regularization = `Regularization()`
	)

Construct RBM.

Parameters

D	number of inputs
H	number of hidden nodes
cdN	number of contrastive divergence steps
stdDev	standard deviation of initial weights
backprop	weights can be finetuned with backprop
regularization	regularization coefficients

virtual OpenANN::RBM::~RBM ( )

virtual

Member Function Documentation

virtual void OpenANN::RBM::backpropagate	(	Eigen::MatrixXd *	ein,
		Eigen::MatrixXd *&	eout,
		bool	backpropToPrevious
	)

virtual

Backpropagation in this layer.

Parameters

ein	pointer to error signal of the higher layer
eout	returns a pointer to error signal of the layer (derivative of the error with respect to the input)
backpropToPrevious	backpropagate errors to previous layers

Implements OpenANN::Layer.

virtual const Eigen::VectorXd& OpenANN::RBM::currentParameters ( )

virtual

Request the current parameters.

Returns: current parameters

Implements OpenANN::Optimizable.

virtual unsigned int OpenANN::RBM::dimension ( )

virtual

Request the number of optimizable parameters.

Returns: number of optimizable parameters

Implements OpenANN::Optimizable.

virtual double OpenANN::RBM::error ( )

virtual

Compute error on training set.

Returns: current error on training set or objective function value

Implements OpenANN::Optimizable.

virtual double OpenANN::RBM::error ( unsigned int n)

virtual

virtual void OpenANN::RBM::errorGradient	(	std::vector< int >::const_iterator	startN,
		std::vector< int >::const_iterator	endN,
		double &	value,
		Eigen::VectorXd &	grad
	)

virtual

Calculates the accumulated gradient and error of given training examples.

Parameters

startN	iterator over index vector
endN	iterator over index vector
value	function value
grad	gradient of the function, lenght must be dimension()

Returns: each row contains the gradient for one training example

Reimplemented from OpenANN::Optimizable.

virtual unsigned int OpenANN::RBM::examples ( )

virtual

Request number of training examples.

Returns: number of training examples

Reimplemented from OpenANN::Optimizable.

virtual void OpenANN::RBM::forwardPropagate	(	Eigen::MatrixXd *	x,
		Eigen::MatrixXd *&	y,
		bool	dropout,
		double *	error = `0`
	)

virtual

Forward propagation in this layer.

Parameters

x	pointer to input of the layer (with bias)
y	returns a pointer to output of the layer
dropout	enable dropout for regularization
error	error value, will be updated with regularization terms

Implements OpenANN::Layer.

virtual Eigen::MatrixXd& OpenANN::RBM::getOutput ( )

virtual

Output after last forward propagation.

Returns: output

Implements OpenANN::Layer.

virtual Eigen::VectorXd OpenANN::RBM::getParameters ( )

virtual

Get the current values of parameters (weights, biases, ...).

Returns: parameters

Implements OpenANN::Layer.

const Eigen::MatrixXd& OpenANN::RBM::getVisibleProbs ( )

Get probabilities of visible units.

Returns: probobilities of visible units

const Eigen::MatrixXd& OpenANN::RBM::getVisibleSample ( )

Get sample from the distribution of the visible units.

Returns: sample from visible units

const Eigen::MatrixXd& OpenANN::RBM::getWeights ( )

Get the current weight matrix.

Returns: weight matrix

virtual Eigen::VectorXd OpenANN::RBM::gradient ( )

virtual

Compute gradient of the error function with respect to the parameters.

Returns: gradient

Implements OpenANN::Optimizable.

virtual Eigen::VectorXd OpenANN::RBM::gradient ( unsigned int n)

virtual

int OpenANN::RBM::hiddenUnits ( )

Get number of hidden units.

Returns: number of hidden units

virtual void OpenANN::RBM::initialize ( )

virtual

Initialize the optimizable parameters.

Implements OpenANN::Optimizable.

virtual OutputInfo OpenANN::RBM::initialize	(	std::vector< double * > &	parameterPointers,
		std::vector< double * > &	parameterDerivativePointers
	)

virtual

Fill in the parameter pointers and parameter derivative pointers.

Parameters

parameterPointers	pointers to parameters
parameterDerivativePointers	pointers to derivatives of parameters

Returns: information about the output of the layer

Implements OpenANN::Layer.

virtual void OpenANN::RBM::initializeParameters ( )

inlinevirtual

Initialize the parameters.

This is usually called before each optimization.

Implements OpenANN::Layer.

virtual Eigen::VectorXd OpenANN::RBM::operator() ( const Eigen::VectorXd & x)

virtual

Make a prediction.

Parameters

x	Input vector.

Returns: Prediction.

Implements OpenANN::Learner.

virtual Eigen::MatrixXd OpenANN::RBM::operator() ( const Eigen::MatrixXd & X)

virtual

Make predictions.

Parameters

X	Each row represents an input vector.

Returns: Each row represents a prediction.

Implements OpenANN::Learner.

virtual bool OpenANN::RBM::providesGradient ( )

virtual

Check if the object provides a gradient of the error function with respect to its parameters.

Returns: does the optimizable provide a gradient?

Implements OpenANN::Optimizable.

virtual bool OpenANN::RBM::providesInitialization ( )

virtual

Check if the object knows how to initialize its parameters.

Returns: does the optimizable object provide a parameter initialization?

Implements OpenANN::Optimizable.

Eigen::MatrixXd OpenANN::RBM::reconstructProb	(	int	n,
		int	steps
	)

Get the probability of the n-th training example after some sampling steps.

Parameters

n	index of training example
steps	number of sampling steps

Returns: probabilities of visible units

void OpenANN::RBM::sampleHgivenV ( )

Sample hidden units given visible units.

void OpenANN::RBM::sampleVgivenH ( )

Sample visible units given hidden units.

virtual void OpenANN::RBM::setParameters ( const Eigen::VectorXd & parameters)

virtual

Set new parameters.

Parameters

parameters new parameters

Implements OpenANN::Optimizable.

virtual void OpenANN::RBM::updatedParameters ( )

inlinevirtual

Generate internal parameters from externally visible parameters.

This is usually called after each parameter update.

Implements OpenANN::Layer.

int OpenANN::RBM::visibleUnits ( )

Get number of visible units.

Returns: number of visible units

The documentation for this class was generated from the following file:

OpenANN/RBM.h

Public Member Functions

Additional Inherited Members

Detailed Description

Constructor & Destructor Documentation

Member Function Documentation