GIT-DPO: Gradient-Informed Token-Weighted Direct Preference Optimization for LLM Alignment

Abstract

Direct Preference Optimization (DPO) has emerged as a popular alternative to Reinforcement Learning from Human Feedback (RLHF) for aligning Large Language Models (LLMs) with human preferences, eliminating the need for explicit reward modeling. However, DPO and most of its variants treat all tokens in a response uniformly during preference optimization, assigning equal optimization pressure to every token regardless of its actual contribution to the preference judgment. This uniform treatment is suboptimal because only a small subset of tokens, such as factual claims, safety-critical terms, or key instruction-following phrases, typically determines the preference label, while the remaining tokens act as noise that dilutes the optimization signal. We propose GIT-DPO, a novel token-level preference optimization framework that combines two complementary innovations: a gradient-based token attribution mechanism that dynamically computes fine-grained importance weights by measuring the sensitivity of the DPO loss to perturbations in each token's log-probability, and a contrastive dual-pair loss that simultaneously pulls the policy toward preferred responses, pushes it away from dispreferred responses, and regularizes low-importance tokens toward the reference model. Extensive experiments on Al-pacaEval 2.0, Arena-Hard, and MT-Bench across three backbone LLMs (Llama-3-8B-Instruct, Mistral-7B-Instruct, Qwen2.5-7B-Instruct) demonstrate that GIT-DPO consistently outperforms nine strong baselines including DPO, SimPO, TDPO, ORPO, KTO, ConfPO, Selective-DPO, and AlignDistil. On Llama-3-8B-Instruct, GIT-DPO achieves a length-controlled win rate of 22.84 percent on AlpacaEval 2.0, outperforming the strongest baseline by 2.74 percentage points, while converging 1.5 times faster than standard DPO with only 5 to 8 percent additional training time per step.