Structured Stochastic Variational Inference

[sharanry]

#775

Key idea:
Relaxes the mean-field approximation (commonly used in Stochastic Variational Inference) to allow arbitrary dependencies between global parameters and local hidden variables, producing better parameter estimates by reducing bias, sensitivity to local optima, and sensitivity to hyperparameters.

Main Reference: https://arxiv.org/abs/1404.4114

No Known Existing Implementation

Key Questions:

1. How does the user define the structure of the variational distribution?

   • One possible way as suggested by @xukai92 is "passing something like a tree (e.g. for the commonly used gdemo it would be (:s=>:x, :m=>:x))". This would mean that we would have to parse this information and build a variational model respecting these dependencies behind the scenes?

   • Another solution might be that the user makes a custom variational distribution(with all the dependencies) and passes it to the VI algorithm which initializing it. Something like as shown here. But creating a custom distribution in this way requires the user to specify many functions which may not all be necessary.

[yebai]

Closed in favour of #774