Generalizations

Importance Sampling ABC

Consider again the Soft-ABC algorithm. Remember that we compute unnormalized weights using a kernel and then we normalize them, similar to how we do in Importance Sampling. Indeed, it turns out we can generalize Soft-ABC to something called Importance Sampling ABC or IS-ABC.

First of all, we need a proposal distribution. Usually one chooses $$q(\theta ,y)=p(y\mid \theta )q(\theta )$$ for some distribution $q(\theta )$. Then the unnormalized importance sampling weights are given by the ratio of the (unnormalized) augmented ABC posterior and the proposal distribution $$\tilde{w}(\theta )=\frac{{\tilde{p}}_{ϵ}(\theta ,y\mid y^{\ast })}{q(\theta ,y)}=\frac{{\tilde{p}}_{ϵ}(y^{\ast }\mid y)p(y\mid \theta )p(\theta )}{p(y\mid \theta )q(\theta )}=\frac{{\tilde{p}}_{ϵ}(y^{\ast }\mid y)p(\theta )}{q(\theta )}$$ The IS-ABC algorithm is given below. Importantly, notice how now we are sampling parameters from $q(\theta )$ rather than from the prior $p(\theta )$.

It is immediately clear that Soft-ABC is just a special case of this where our proposal distribution is $p(y\mid \theta )p(\theta )$.

You can play around with IS-ABC applied to the Two Moons example here. Notice that it may take a while to run.

Generalized Rejection ABC

One can generalize Rejection-ABC to a likelihood-free rejection sampler. The target distribution is still the augmented ABC posterior $$p_{ϵ}(\theta ,y\mid y^{\ast })=p_{ϵ}(y^{\ast }\mid y)p(y\mid \theta )p(\theta )$$ and, as typical in rejection sampling, we require a proposal density $q(\theta ,s)$ satisfying the following bound $${\tilde{p}}_{ϵ}(\theta ,y\mid y^{\ast })\le Mq(\theta ,s)\text{for some }M>0$$ One can then sample from $q(\theta ,s)$ and accept draws with probability $$\frac{{\tilde{p}}_{ϵ}(\theta ,y\mid y^{\ast })}{Mq(\theta ,s)}$$ In particular, here we choose the following proposal distribution $$q(\theta ,s)=p(y\mid \theta )q(\theta )$$ for some $q(\theta )$ satisfying the bound above. Of course, the reason of this choice of proposal distribution is so that the intractable likelihood cancels out in the acceptance probability $$\frac{{\tilde{p}}_{ϵ}(\theta ,y\mid y^{\ast })}{Mq(\theta ,s)}=\frac{{\tilde{p}}_{ϵ}(y^{\ast }\mid y)p(y\mid \theta )p(\theta )}{Mp(y\mid \theta )q(\theta )}=\frac{{\tilde{p}}_{ϵ}(y^{\ast }\mid y)p(\theta )}{Qq(\theta )}$$ The the generalized rejection-ABC algorithm is given below.

It is well-known in Rejection sampling that the optimal value for the constant $M$ is $$M^{\ast }=\underset{\theta ,y}{max}\frac{{\tilde{p}}_{ϵ}(\theta ,y\mid y^{\ast })}{p(y\mid \theta )q(\theta )}=\underset{\theta ,y}{max}\frac{{\tilde{p}}_{ϵ}(y^{\ast }\mid y)p(\theta )}{q(\theta )}=\underset{y}{max}{\tilde{p}}_{ϵ}(y^{\ast }\mid y)\underset{\theta }{max}\frac{p(\theta )}{q(\theta )}$$