Direct Preference Optimization

From the Bayesian perspective, we know that the optimal policy is given by $${\pi }_{\varphi }^{\ast }(x)=\frac{1}{Z}{\pi }_0(x)\exp (r_{\varphi }(x)/\beta ),Z=\int {\pi }_0(x^{\prime })\exp (r_{\varphi }(x^{\prime })/\beta )d{x}^{\prime }$$ The key insight behind Direct Preference Optimization is that one re-write this to express the reward function as follows $$r_{\varphi }(y\mid x)=\beta \log \left(\frac{{\pi }_{\varphi }^{\ast }(y\mid x)}{{\pi }_0(y\mid x)}\right)+\beta \log Z.$$ This is essentially just another parametrization of the reward model. Now, recall that the Bradley-Terry model depends only on the difference in reward and therefore plugging this parametrization (and using our policy ${\pi }_{\theta }$ rather than the actual optimal ${\pi }_{\varphi }^{\ast }$) there the normalizing constant cancels out $$p_{\theta }^{\ast }(y_w\succ y_l\mid x)=\text{sigmoid}(\beta \log \left(\frac{{\pi }_{\theta }(y_w\mid x)}{{\pi }_0(y_w\mid x)}\right)-\beta \log \left(\frac{{\pi }_{\theta }(y_l\mid x)}{{\pi }_0(y_l\mid x)}\right))$$ We can therefore perform maximum likelihood directly on the policy. Our loss function therefore becomes $${\mathcal{L}}_{\text{DPO}}(\theta )=-{\mathbb{E}}_{(x,y_w,y_l)\sim {\mathcal{D}}_{\text{preferences}}}[\log \left\{\text{sigmoid}(\beta \log \left(\frac{{\pi }_{\theta }(y_w\mid x)}{{\pi }_0(y_w\mid x)}\right)-\beta \log \left(\frac{{\pi }_{\theta }(y_l\mid x)}{{\pi }_0(y_l\mid x)}\right))\right\}]$$