Bayesian Perspective

The optimization problem of PPO can be rewritten from a Bayesian perspective $$\begin{array}{rl}\underset{\theta }{max}\{{\mathbb{E}}_{{\pi }_{\theta }}[r_{\varphi }(X,Y)]-\beta \text{KL}({\pi }_{\theta }\parallel {\pi }_0)\}& =\underset{\theta }{max}{\mathbb{E}}_{{\pi }_{\theta }}[r_{\varphi }(X,Y)-\beta \log {\pi }_{\theta }(X)+\beta \log {\pi }_0(X)]\\ & =\underset{\theta }{max}{\mathbb{E}}_{{\pi }_{\theta }}[(\frac{r_{\varphi }(X,Y)}{\beta }+\log {\pi }_0(X))-\beta \log {\pi }_{\theta }(X)]\\ & =\underset{\theta }{min}\text{KL}({\pi }_{\theta }\parallel {\pi }_{\varphi }^{\ast })\end{array}$$ where we have defined the distribution $${\pi }_{\varphi }^{\ast }(x)=\frac{1}{Z}{\pi }_0(x)\exp (r_{\varphi }(x)/\beta ),Z=\int {\pi }_0(x^{\prime })\exp (r_{\varphi }(x^{\prime })/\beta )d{x}^{\prime }$$ which can be interpreted as a posterior distribution with prior ${\pi }_0$ and evidence represented by the reward model $r_{\varphi }$. This Bayesian perspective was fully fleshed out in this paper. Notice that then the task of performing RLHF can be cast as variational inference with proposal ${\pi }_{\theta }$ and target ${\pi }_{\varphi }^{\ast }$.