Classic Importance Sampling

Suppose that our aim is to compute the posterior expectation $${\mathbb{E}}_{p(\mathit{\theta }\mid \mathbf{y})}[h(\mathit{\theta })]=\int h(\mathit{\theta })p(\mathit{\theta }\mid \mathbf{y})d\mathit{\theta }$$ of some function $h(\mathit{\theta })$ with respect to a posterior distribution

$$\begin{array}{r}p(\mathit{\theta }\mid \mathbf{y})=\frac{\tilde{p}(\mathit{\theta }\mid \mathbf{y})}{Z_p}\text{and}{Z}_p=\int \tilde{p}(\mathit{\theta }\mid \mathbf{y})d\mathit{\theta }.\end{array}$$

Suppose also that it is impractical to sample directly from $p(\mathit{\theta }\mid \mathbf{y})$. In Importance Sampling we choose an importance distribution $q(\mathit{\theta })$ from which it is easier to sample from, and rewrite the expectation above as an expectation with respect to $q(\mathit{\theta })$ so that we can use its samples instead $$\begin{array}{rl}{\mathbb{E}}_{p(\mathit{\theta }\mid \mathbf{y})}[h(\mathit{\theta })]& =\int h(\mathit{\theta })p(\mathit{\theta }\mid \mathbf{y})d\mathit{\theta }\\ & =\int h(\mathit{\theta })\frac{\tilde{p}(\mathit{\theta }\mid \mathbf{y})}{Z_p}d\mathit{\theta }\\ & =\frac{1}{Z_p}\int h(\mathit{\theta })\frac{\tilde{p}(\mathit{\theta }\mid \mathbf{y})}{q(\mathit{\theta })}q(\mathit{\theta })d\mathit{\theta }\\ & =\frac{\int h(\mathit{\theta })\frac{\tilde{p}(\mathit{\theta }\mid \mathbf{y})}{q(\mathit{\theta })}q(\mathit{\theta })d\mathit{\theta }}{\int \frac{\tilde{p}(\mathit{\theta }\mid \mathbf{y})}{q(\mathit{\theta })}q(\mathit{\theta })d\mathit{\theta }}\\ & =\frac{{\mathbb{E}}_{q(\mathit{\theta })}[\frac{\tilde{p}(\mathit{\theta }\mid \mathbf{y})}{q(\mathit{\theta })}h(\mathit{\theta })]}{{\mathbb{E}}_{q(\mathit{\theta })}\left[\frac{\tilde{p}(\mathit{\theta }\mid \mathbf{y})}{q(\mathit{\theta })}\right]}.\end{array}$$ Now we approximate this expectation using Monte Carlo using ${\mathit{\theta }}^{(1)},\dots ,{\mathit{\theta }}^{(N)}\sim q(\mathit{\theta })$ $$\begin{array}{rl}{\mathbb{E}}_{p(\mathit{\theta }\mid \mathbf{y})}[h(\mathit{\theta })]& =\frac{{\mathbb{E}}_{q(\mathit{\theta })}[\frac{\tilde{p}(\mathit{\theta }\mid \mathbf{y})}{q(\mathit{\theta })}h(\mathit{\theta })]}{{\mathbb{E}}_{q(\mathit{\theta })}\left[\frac{\tilde{p}(\mathit{\theta }\mid \mathbf{y})}{q(\mathit{\theta })}\right]}\\ & \approx \frac{\frac{1}{N}\sum _{i=1}^N\frac{\tilde{p}({\mathit{\theta }}^{(i)}\mid \mathbf{y})}{q({\mathit{\theta }}^{(i)})}h({\mathit{\theta }}^{(i)})}{\frac{1}{N}\sum _{i=1}^N\frac{\tilde{p}({\mathit{\theta }}^{(i)}\mid \mathbf{y})}{q({\mathit{\theta }}^{(i)})}}\\ & =\sum _{i=1}^N\left[\frac{\frac{\tilde{p}({\mathit{\theta }}^{(i)}\mid \mathbf{y})}{q({\mathit{\theta }}^{(i)})}}{\sum _{j=1}^N\frac{\tilde{p}({\mathit{\theta }}^{(j)}\mid \mathbf{y})}{q({\mathit{\theta }}^{(j)})}}\right]h({\mathit{\theta }}^{(i)})\\ & =\sum _{i=1}^N\left[\frac{\tilde{w}({\mathit{\theta }}^{(i)})}{\sum _{j=1}^N\tilde{w}({\mathit{\theta }}^{(j)})}\right]h({\mathit{\theta }}^{(i)})\\ & =\sum _{i=1}^{N}w({\mathit{\theta }}^{(i)})h({\mathit{\theta }}^{(i)})\end{array}$$

where we have defined $w({\mathit{\theta }}^{(i)})$ and $\tilde{w}({\mathit{\theta }}^{(i)})$ are called normalized importance weights and unnormalized importance weights respectively and are defined below $$\tilde{w}({\mathit{\theta }}^{(i)})=\frac{\tilde{p}({\mathit{\theta }}^{(i)}\mid \mathbf{y})}{q({\mathit{\theta }}^{(i)})}\text{and}w({\mathit{\theta }}^{(i)})=\frac{\tilde{w}({\mathit{\theta }}^{(i)})}{\sum _{j=1}^N\tilde{w}({\mathit{\theta }}^{(j)})}$$

Properties of Classic Importance Sampling

• For the variance of the estimator of ${\mathbb{E}}_{p(\mathit{\theta }\mid \mathbf{y})}[h(\mathit{\theta })]$ to be finite we require $\text{supp}(q)\supset \text{supp}(p)$.
• Notice that the vector of weights $\mathbf{w}=(w({\mathit{\theta }}^{(1)}),\dots ,w({\mathit{\theta }}^{(N)}){)}^{\mathrm{\top }}$ tells us how well $q(\mathit{\theta })$ fits $p(\mathit{\theta }\mid \mathbf{y})$. In particular when $q\equiv p$ we have $\mathbf{w}={(\frac{1}{N},\dots ,\frac{1}{N})}^{\mathrm{\top }}$, i.e. a vector of uniform weights. So the more uniform the weights, the better the fit. In practice, there will be only few dominant weights, and if $q$ badly fits $p$ then the vast majority of weights will be neglegible, and one or two weights will dominate.