Multivariate Normal as an Exponential Family Distribution

Exponential Family of Distributions

A density $f(\mathbf{x})$ belongs to the exponential family of distributions if we can write it as $$f(\mathbf{x};\mathit{\theta })=\exp \{⟨\mathit{\theta },\varphi (\mathbf{x})⟩-A(\mathit{\theta })\}$$ we call $\mathit{\theta }$ its natural parameters, while we call ${\mathbb{E}}_f[\varphi (\mathbf{X})]$ its mean parameters.

Multivariate Normal Distribution

A pdf $f$ is a multivariate normal distribution if $$f(\mathbf{x})=(2\pi {)}^{-\frac{d}{2}}\text{det}(\mathbf{\Sigma }{)}^{-\frac{1}{2}}\exp \{-\frac{1}{2}(\mathbf{x}-\mathit{\mu }{)}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}(\mathbf{x}-\mathit{\mu })\}$$

This can be rearranged as $$f(\mathbf{x})=\exp \{{\mathbf{x}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathit{\mu }-\frac{1}{2}{\mathbf{x}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{x}-\frac{1}{2}[d\log 2\pi +\log |\mathbf{\Sigma }|+{\mathit{\mu }}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathit{\mu }]\}$$

Frobenius Inner Product

Notice that we can write the second term as $$-\frac{1}{2}{\mathbf{x}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{x}=-\frac{1}{2}\sum _{k=1}^d\sum _{j=1}^d{x}_k{\mathrm{\Sigma }}_{kj}^{-1}{x}_j$$ similarly, the following expression can be written in the same way $$\begin{array}{rl}\text{tr}[-\frac{1}{2}{\mathbf{\Sigma }}^{-1}\mathbf{x}{\mathbf{x}}^{\mathrm{\top }}]& =-\frac{1}{2}\text{tr}\left[\left(\begin{array}{ccc}{\mathrm{\Sigma }}_{11}^{-1}& \cdots & {\mathrm{\Sigma }}_{1d}^{-1}\\ ⋮& \ddots & ⋮\\ {\mathrm{\Sigma }}_{d1}^{-1}& \cdots & {\mathrm{\Sigma }}_{dd}^{-1}\end{array}\right)\left(\begin{array}{ccc}{x}_1^2& \cdots & x_1{x}_d\\ ⋮& \ddots & ⋮\\ x_d{x}_1& \cdots & x_d^2\end{array}\right)\right]\\ & =-\frac{1}{2}\text{tr}\left[\left(\begin{array}{ccc}\sum _{j=1}^d{\mathrm{\Sigma }}_{1j}^{-1}{x}_j{x}_1& \cdots & \sum _{j=1}{\mathrm{\Sigma }}_{1j}^{-1}{x}_j{x}_d\\ ⋮& \ddots & ⋮\\ \sum _{j=1}^d{\mathrm{\Sigma }}_{dj}^{-1}{x}_j{x}_1& \cdots & \sum _{j=1}^d{\mathrm{\Sigma }}_{dj}^{-1}{x}_j{x}_d\end{array}\right)\right]\\ & =-\frac{1}{2}\sum _{k=1}^d\sum _{j=1}^d{x}_k{\mathrm{\Sigma }}_{kj}^{-1}{x}_j\end{array}$$ Now notice that this is nothing but the Frobenius inner product between two real and symmetric matrices (which can be written both in terms of a trace and in terms of $\text{vec}$-torizing operations) $$\begin{array}{rl}{⟨-\frac{1}{2}{\mathbf{\Sigma }}^{-1},\mathbf{x}{\mathbf{x}}^{\mathrm{\top }}⟩}_F& =\text{tr}(-\frac{1}{2}{\mathbf{\Sigma }}^{-1}\mathbf{x}{\mathbf{x}}^{\mathrm{\top }})\\ & =\text{vec}{(-\frac{1}{2}{\mathbf{\Sigma }}^{-1})}^{\mathrm{\top }}\text{vec}\left(\mathbf{x}{\mathbf{x}}^{\mathrm{\top }}\right)\end{array}$$ where the vectorized operation for an $n\times m$ matrix $A$ simply stacks the rows one at a time to create a $(nm)\times 1$ vector $$\text{vec}[A]=\text{vec}\left[\left(\begin{array}{ccc}{a}_{11}& \cdots & a_{1m}\\ ⋮& \ddots & ⋮\\ a_{n1}& \cdots & a_{nm}\end{array}\right)\right]=\left(\begin{array}{c}{a}_{11}\\ ⋮\\ a_{1m}\\ ⋮\\ a_{nm}\end{array}\right)$$

this allows us to write the pdf as $$f(\mathbf{x})=\exp \{⟨{\mathbf{\Sigma }}^{-1}\mathit{\mu },\mathbf{x}⟩+⟨\text{vec}(-\frac{1}{2}{\mathbf{\Sigma }}^{-1}),\text{vec}\left(\mathbf{x}{\mathbf{x}}^{\mathrm{\top }}\right)⟩-\frac{1}{2}[d\log 2\pi +\log |\mathbf{\Sigma }|+{\mathit{\mu }}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathit{\mu }]\}$$

Natural Parameters of a Multivariate Normal Distribution

Since ${\mathbf{\Sigma }}^{-1}\mathit{\mu }$ can be written as $${\mathbf{\Sigma }}^{-1}\mathit{\mu }=\left(\begin{array}{c}\sum _{j=1}^d{\mathrm{\Sigma }}_{1j}^{-1}{\mu }_j\\ ⋮\\ \sum _{j=1}^d{\mathrm{\Sigma }}_{dj}^{-1}{\mu }_j\end{array}\right)$$ the natural parameters are given by $$\mathit{\theta }=\left(\begin{array}{c}{\mathbf{\Sigma }}^{-1}\mathit{\mu }\\ \text{vec}[-\frac{1}{2}{\mathbf{\Sigma }}^{-1}]\end{array}\right)={\left(\begin{array}{c}\sum _{j=1}^d{\mathrm{\Sigma }}_{1j}^{-1}{\mu }_j\\ ⋮\\ \sum _{j=1}^d{\mathrm{\Sigma }}_{dj}^{-1}{\mu }_j\\ -\frac{1}{2}{\mathrm{\Sigma }}_{11}^{-1}\\ ⋮\\ -\frac{1}{2}{\mathrm{\Sigma }}_{1d}^{-1}\\ ⋮\\ -\frac{1}{2}{\mathrm{\Sigma }}_{dd}^{-1}\end{array}\right)}_{(d+d^2)\times 1}$$

In a similar fashion, we can find the sufficient statistics as $$\varphi (\mathbf{x})=\left(\begin{array}{c}\mathbf{x}\\ \text{vec}(\mathbf{x}{\mathbf{x}}^{\mathrm{\top }})\end{array}\right)={\left(\begin{array}{c}{x}_1\\ ⋮\\ x_d\\ x_1^2\\ ⋮\\ x_1{x}_d\\ ⋮\\ x_d^2\end{array}\right)}_{(d+d^2)\times 1}$$

This gives the complete expression for the multivariate normal distribution as part of the Exponential Family of distributions

$$\begin{array}{rl}f(\mathbf{x})& =\exp \{⟨\theta ,\varphi (\mathbf{x})⟩-A(\mathit{\theta })\}\\ & =\exp \{{(\sum _{j=1}^d{\mathrm{\Sigma }}_{1j}^{-1}{\mu }_j,\cdots ,\sum _{j=1}^d{\mathrm{\Sigma }}_{dj}^{-1}{\mu }_j,-\frac{1}{2}{\mathrm{\Sigma }}_{11}^{-1},\cdots ,-\frac{1}{2}{\mathrm{\Sigma }}_{1d}^{-1},\cdots ,-\frac{1}{2}{\mathrm{\Sigma }}_{dd}^{-1})}^{\mathrm{\top }}\left(\begin{array}{c}{x}_1\\ ⋮\\ x_d\\ x_1^2\\ ⋮\\ x_1{x}_d\\ ⋮\\ x_d^2\end{array}\right)-\frac{1}{2}[d\log 2\pi +\log |\mathbf{\Sigma }|+{\mathit{\mu }}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathit{\mu }]\}\end{array}$$

Mean Parameters of a Multivariate Normal Distribution

Remember that the expected value of a matrix or a vector is taken element-wise, so that if we were to compute $$\begin{array}{rl}(\mathbb{E}[\mathbf{x}{\mathbf{x}}^{\mathrm{\top }}]{)}_{ij}& =\mathbb{E}[(\mathbf{x}{\mathbf{x}}^{\mathrm{\top }}{)}_{ij}]\\ & =\mathbb{E}[x_i{x}_j]\\ & =\text{cov}(x_i,x_j)+\mathbb{E}[x_i]\mathbb{E}[x_j]\\ & ={\mathrm{\Sigma }}_{ij}+{\mu }_i{\mu }_j\end{array}$$ This means that taking the expected value of our vectorized matrix $\text{vec}(\mathbf{x}{\mathbf{x}}^{\mathrm{\top }})$ is equivalent to vectorizing the expected value of $\mathbf{x}{\mathbf{x}}^{\mathrm{\top }}$ $$\begin{array}{rl}\mathbb{E}[\text{vec}(\mathbf{x}{\mathbf{x}}^{\mathrm{\top }})]& =\mathbb{E}\left[\left(\begin{array}{c}{x}_1^2\\ ⋮\\ x_1{x}_d\\ ⋮\\ x_d^2\end{array}\right)\right]\\ & =\left(\begin{array}{c}\mathbb{E}\left[x_1^2\right]\\ ⋮\\ \mathbb{E}\left[x_1{x}_d\right]\\ ⋮\\ \mathbb{E}\left[x_d^2\right]\end{array}\right)\\ & =\left(\begin{array}{c}{\mathrm{\Sigma }}_{11}+{\mu }_1^2\\ ⋮\\ {\mathrm{\Sigma }}_{1d}+{\mu }_1{\mu }_d\\ ⋮\\ {\mathrm{\Sigma }}_{dd}+{\mu }_d^2\end{array}\right)\\ & =\text{vec}\left(\mathbb{E}\left[\mathbf{x}{\mathbf{x}}^{\mathrm{\top }}\right]\right)\end{array}$$ Finally, the mean parameters are given by $$\begin{array}{rl}\mathbb{E}[\varphi (\mathbf{x})]& =\mathbb{E}\left[\left(\begin{array}{c}\mathbf{x}\\ \text{vec}(\mathbf{x}{\mathbf{x}}^{\mathrm{\top }})\end{array}\right)\right]\\ & =\left(\begin{array}{c}\mathit{\mu }\\ \text{vec}(\mathbf{\Sigma }+\mathit{\mu }{\mathit{\mu }}^{\mathrm{\top }})\end{array}\right)\end{array}$$