Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

=Made changed to RBMs (and DBNs) to allow for gaussian units #128

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

oxinabox
Copy link

Motivation

Normalising is a common way to pre-process continuously valued (unbounded) inputs.
'Natural' Guassian process also exist.
Both these input types will not work well with the traditional Bernoulli input layers of an RBM.

The Gaussian Bernoulli RBM was made to deal with this.
Numerous papers talk about it, including: "A Practical Guide to Training Restricted Boltzmann Machines" Geoffrey Hinton 2010, and "Greedy layer-wise training of deep networks." Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle 2007 - Further details of DBNs, including Gaussian Bernoulli unit

Changes:

RBMs now have a type,
either 'gb' for Gaussian Bernoulli (AKA Gaussian Binary),
or 'bb' for Bernoulli-Bernoulli (AKA Binary-Binary).
If not specified defaults to 'bb'

  • Training and encoding algorithm was added.
  • Test case was added (it performs normally with ~0.05% error)
  • RBM Setup was moved to its own file to clean the code and facilitate the changes
  • Batch division was made into a warning instead of an error.

Limitations

Support was only added for inputs with Unit Variance. NonUnit variance Gasssian inputs are possible but annoying, and would require more restructuring.

(Credit to the Signals and Information Processing Lab of the School of Electrical, Electronic and Computer Engineering, with-in the University of Western Australia, who are (almost directly) paying me to make these changes (and who are happy to have them pushed back upstream).)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant