Objective Function and Update Equations

Data Log-Likelihood

Of course, we know what $p(x)$ looks like, it’s just the pdf of a normal distribution. However, in practice, this could be a very complicated density and in that case, we can use the univariate change of variable formula to find its pdf

$$\log p(x)=\log p(g^{-1}(x))+\log |\frac{\partial }{\partial x}{g}^{-1}(x)|.$$

The first term can be found by using the pdf for a standard normal distribution $$\log p(z)=-\frac{1}{2}\log (2\pi )-\frac{z^2}{2}$$ and plugging in the inverse transformation $g^{-1}(x)=x-\mu$ $$\log p(g^{-1}(x))=-\frac{1}{2}\log (2\pi )-\frac{(x-\mu {)}^2}{2}.$$ For the second term, we need to compute the derivative of $g^{-1}(x)$ with respect to $x$ $$\frac{\partial }{\partial x}{g}^{-1}(x)=1$$ and remember that $\log (1)=0$ to obtain:

$$\begin{array}{rl}\log p(x)& =-\frac{1}{2}\log (2\pi )-\frac{(x-\mu {)}^2}{2}.\end{array}$$

Since the samples are i.i.d. we can estimate the log-likelihood very easily. $$\begin{array}{rl}\sum _{i=1}^n\log p(x^{(i)})& =-\frac{n}{2}\log (2\pi )-\frac{1}{2}\sum _{i=1}^n(x^{(i)}-\mu {)}^2\end{array}$$

Our objective function to minimize is then the negative log-likelihood. Often one uses the negative average log-likelihood instead, because this leads to more consistent gradients across different dataset sizes, you can read more about it here

$$\frac{1}{n}\sum _{i=1}^n\log p(x^{(i)})=-\frac{1}{2}\log (2\pi )-\frac{1}{2n}\sum _{i=1}^n(x^{(i)}-\mu {)}^2$$

Log-Likelihood Gradient Estimates

We can now compute the gradient of the negative average log-likelihood with respect to $\mu$

$$\begin{array}{rl}\frac{\partial }{\partial \mu }\frac{1}{n}\sum _{i=1}^n\log p(x^{(i)})& =\frac{1}{n}\sum _{i=1}^n{x}^{(i)}-\mu \end{array}$$

These can be easily computed since we know $n$, and $\mu$ so we compute $x^{(i)}=z^{(i)}+\mu$.

Mean Update Equation

We can then update our mean using gradient descent with step size $\gamma$: $$\begin{array}{rl}{\mu }_t& ⟵{\mu }_{t-1}-\gamma [\frac{1}{n}\sum _{i=1}^n{x}^{(i)}-{\mu }_{t-1}]\end{array}$$