Variational Inference Design

[theogf]

Here is to discuss about the design concerning variational inference methods.

So far in AugmentedGaussianProcesses.jl things are done this way:

• There are two functions ∇E_μ and ∇E_Σ which return the gradient of the expected log likelihood given mu and Sigma (the second one returns an arrays as usually you don't need the covariance terms of Sigma).
• Then the gradient given the ELBO is actually given the natural gradient because there is the relationship nat. grad. of ELBO given natural parameters (eta_1 (Sigma^-1 mu), eta_2 (-0.5Sigma^-1)) is equal to normal grad. of ELBO given normal parameters (mu, Sigma).
  • With the augmentation procedure it's possible to get analytical gradients and set the gradient to 0 and it becomes a coordinate ascent update (or a partial one with a learning rate for the stochastic case)
  • Without it, I just use a natural grad. ascent scheme.
• Then natural parameters are mapped back to normal parameters.

[theogf]

Note that for the non-analytic case and for 1-D likelihood (needing only one GP), one can use the Opper and Archambeau (2009) equality :
grad_mu = expec[dlogp/df]
grad_sig = 0.5 expec[d^2logp/d^2f]