Multivariate Normal Distribution in the Exponential Family

Remember from a previous blog post that a pdf $p(\mathbf{x})=\mathcal{N}(\mathbf{x};\mathit{\mu },\mathbf{\Sigma })$ can be written as $$p(\mathbf{x})=\exp \{⟨{\mathbf{\Sigma }}^{-1}\mathit{\mu },\mathbf{x}⟩+⟨\text{vec}(-\frac{1}{2}{\mathbf{\Sigma }}^{-1}),\text{vec}\left(\mathbf{x}{\mathbf{x}}^{\mathrm{\top }}\right)⟩-\frac{1}{2}[d\log 2\pi +\log |\mathbf{\Sigma }|+{\mathit{\mu }}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathit{\mu }]\}$$

Following the work of Barthelmé, Chopin, and Cottet (2015) we can relabel the expression above as follows $$p(\mathbf{x})\propto \exp \{⟨\mathbf{r},\mathbf{x}⟩+⟨\text{vec}(-\frac{1}{2}\mathbf{Q}),\text{vec}\left(\mathbf{x}{\mathbf{x}}^{\mathrm{\top }}\right)⟩\}$$ where we have defined $$\mathbf{r}:={\mathbf{\Sigma }}^{-1}\mathit{\mu }\text{and}\mathbf{Q}:={\mathbf{\Sigma }}^{-1}$$

Target Distribution

We assume the target distribution is intractable, but can be factorized into a product of $K+1$ terms $$f(\mathit{\gamma })\propto \prod _{k=0}^K{f}_k({\mathit{\gamma }}_k)$$ where usually

• $f(\mathit{\gamma })=p(\mathit{\gamma }\mid \mathbf{x})$ is a posterior distribution.
• $f_0({\mathit{\gamma }}_0)=p(\mathit{\gamma };{\mathit{\theta }}_0)$ is a prior distribution.
• $f_1({\mathit{\gamma }}_1),\dots ,f_K({\mathit{\gamma }}_k)$ are likelihood terms, which are intractable.

From here onwards we assume that the prior distribution $f_0(\mathit{\gamma })$ is a multivariate gaussian distribution $\mathcal{N}(\mathit{\gamma };{\mathbf{r}}_0,{\mathbf{Q}}_0)$ (with natural parameters ${\mathit{\theta }}_0={({\mathbf{r}}_0,-\frac{1}{2}{\mathbf{Q}}_0)}^{\mathrm{\top }}$). $$f_0(\mathit{\gamma })=p(\mathit{\gamma };{\mathbf{r}}_0,{\mathbf{Q}}_0)=\propto \exp \{⟨{\mathbf{r}}_0,\mathbf{x}⟩+⟨\text{vec}(-\frac{1}{2}{\mathbf{Q}}_0),\text{vec}\left(\mathbf{x}{\mathbf{x}}^{\mathrm{\top }}\right)⟩\}$$

Note: The parameters ${\mathit{\gamma }}_0,\dots ,{\mathit{\gamma }}_k$ are not the components of $\mathit{\gamma }$, they are different parameters.

Global Approximation

The global approximation is defined as $$g(\mathit{\theta })\propto \prod _{k=0}^K{g}_k({\mathit{\theta }}_k)$$ where we set the first term to be equal to the (tractable) prior, which is a multivariate gaussian distribution $$g_0(\mathit{\theta })\propto \exp \{⟨{\mathbf{r}}_0,\mathbf{x}⟩+⟨\text{vec}(-\frac{1}{2}{\mathbf{Q}}_0),\text{vec}\left(\mathbf{x}{\mathbf{x}}^{\mathrm{\top }}\right)⟩\}$$ Furthermore, we assume that each factor $g_k({\mathit{\theta }}_k)$ for $k=1,\dots ,K$ also follows a Multivariate Normal Distribution with natural parameter ${\mathit{\theta }}_k=({\mathbf{r}}_k,{\mathbf{Q}}_k{)}^{\mathrm{\top }}$. Luckily, the product of Gaussians is again a Gaussian distribution. In particular we have $$\begin{array}{rl}g(\mathit{\theta })& \propto \prod _{k=0}^K\exp \{{\mathbf{r}}_k^{\mathrm{\top }}\mathbf{x}+\text{vec}{(-\frac{1}{2}{\mathbf{Q}}_k)}^{\mathrm{\top }}\text{vec}\left(\mathbf{x}{\mathbf{x}}^{\mathrm{\top }}\right)\}\\ & =\exp \{{\left(\sum _{k=0}^K{\mathbf{r}}_k\right)}^{\mathrm{\top }}\mathbf{x}+\text{vec}{(-\frac{1}{2}\sum _{k=0}^K{\mathbf{Q}}_k)}^{\mathrm{\top }}\text{vec}\left(\mathbf{x}{\mathbf{x}}^{\mathrm{\top }}\right)\}\\ & :=\exp \{{\mathbf{r}}^{\mathrm{\top }}\mathbf{x}+\text{vec}{(-\frac{1}{2}\mathbf{Q})}^{\mathrm{\top }}\text{vec}\left(\mathbf{x}{\mathbf{x}}^{\mathrm{\top }}\right)\}\end{array}$$ where we have defined the global natural parameters to be $$\mathbf{r}:=\sum _{k=0}^K{\mathbf{r}}_k\text{and}\mathbf{Q}:=\sum _{k=0}^K{\mathbf{Q}}_k$$ In other words, the natural parameters of the global approximation are found by summing the natural parameters of all the sites.

Note: The parameters ${\mathit{\theta }}_0,\dots ,{\mathit{\theta }}_k$ are not the components of $\mathit{\theta }$, they are different parameters.

Cavity Distribution

The cavity distribution at the $k^{\text{th}}$ site is given by the product of all but the $k^{\text{th}}$ Multivariate Gaussian. This means that we can write the cavity distribution as $$g_{-k}(\mathit{\theta }-{\mathit{\theta }}_k)\propto \exp \{{(\mathbf{r}-{\mathbf{r}}_k)}^{\mathrm{\top }}\mathbf{x}+\text{vec}{(-\frac{1}{2}(\mathbf{Q}-{\mathbf{Q}}_k))}^{\mathrm{\top }}\text{vec}\left(\mathbf{x}{\mathbf{x}}^{\mathrm{\top }}\right)\}$$ In other words, the natural parameters of the cavity distribution are found by taking the difference between the global natural parameters and the natural parameters of the $k^{\text{th}}$ site.

Tilted Distribution

The tilted distribution, also called pseudo-posterior, is found by multiplying the cavity distribution by the $k^{\text{th}}$ local likelihood term. In other words, the tilted distribution is a (pseudo-)posterior where we use the cavity distribution as a (pseudo-)prior and the single local likelihood term as the likelihood.

In general, computing moments of this distribution will be intractable, however we how that calculating moments of this distribution will be easier than calculating moments of the entire target distribution. $$\begin{array}{rl}{g}_{\mathrm{\setminus }k}({\tilde{\mathit{\gamma }}}_k)& \propto f_k({\mathit{\gamma }}_k)\exp \{{(\mathbf{r}-{\mathbf{r}}_k)}^{\mathrm{\top }}\mathbf{x}+\text{vec}{(-\frac{1}{2}(\mathbf{Q}-{\mathbf{Q}}_k))}^{\mathrm{\top }}\text{vec}\left(\mathbf{x}{\mathbf{x}}^{\mathrm{\top }}\right)\}\end{array}$$

Gaussian Expectation Propagation Updates