=Made changed to RBMs (and DBNs) to allow for gaussian units #128
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Normalising is a common way to pre-process continuously valued (unbounded) inputs.
'Natural' Guassian process also exist.
Both these input types will not work well with the traditional Bernoulli input layers of an RBM.
The Gaussian Bernoulli RBM was made to deal with this.
Numerous papers talk about it, including: "A Practical Guide to Training Restricted Boltzmann Machines" Geoffrey Hinton 2010, and "Greedy layer-wise training of deep networks." Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle 2007 - Further details of DBNs, including Gaussian Bernoulli unit
Changes:
RBMs now have a type,
either 'gb' for Gaussian Bernoulli (AKA Gaussian Binary),
or 'bb' for Bernoulli-Bernoulli (AKA Binary-Binary).
If not specified defaults to 'bb'
Limitations
Support was only added for inputs with Unit Variance. NonUnit variance Gasssian inputs are possible but annoying, and would require more restructuring.
(Credit to the Signals and Information Processing Lab of the School of Electrical, Electronic and Computer Engineering, with-in the University of Western Australia, who are (almost directly) paying me to make these changes (and who are happy to have them pushed back upstream).)