preference-based rl

Preference-Based RL is a reinforcement learning paradigm where the reward signal comes from human preferences over trajectory pairs — instead of numeric rewards, a human evaluator compares two behaviors and indicates which is preferred, and a reward model is learned from these comparisons.

Preference Learning Pipeline

• Query: Present the human with two trajectory segments (\sigma_1, \sigma_2).
• Label: Human indicates preference: \sigma_1 succ \sigma_2, \sigma_2 succ \sigma_1, or roughly equal.
• Reward Model: Train R(s,a) such that P(\sigma_1 succ \sigma_2) = \frac{\exp(\sum R(\sigma_1))}{\exp(\sum R(\sigma_1)) + \exp(\sum R(\sigma_2))} (Bradley-Terry model).
• RL: Optimize policy using the learned reward model via standard RL (PPO, SAC, etc.).

Why It Matters

• Reward-Free: No need to hand-craft reward functions — preferences define the objective implicitly.
• Scalable: Preferences are faster to provide than demonstrations — binary comparison is cognitively easy.
• Active Queries: Active learning selects the most informative trajectory pairs to query — minimizes human effort.

Preference-Based RL is learning rewards from comparisons — using human preferences over behaviors to automatically derive reward functions.